I actually have a copy of the data from Tuesday at 2:30pm. If you need any information out of the results page, just let me know.
Julie On Thu, Oct 15, 2009 at 10:58 AM, Ojan Vafai <o...@google.com> wrote: > I put some more thought into this. Given that we only store a month's worth > of data, it's not worth doing backups. > > Keeping around all the data (maybe a year's worth?) would be awesome > though. I actually think that would not be too much work and would add value > to the dashboard. At that point, doing backups seems more worthwhile. > > I'm happy to walk someone through how to make this happen. It really would > not be a lot of work if you have a workable knowledge of Python and JS. > > Ojan > > > On Wed, Oct 14, 2009 at 5:15 PM, Nicolas Sylvain <nsylv...@chromium.org>wrote: > >> On Wed, Oct 14, 2009 at 4:53 PM, Ojan Vafai <o...@google.com> wrote: >> >>> The data is stored in a single file per bot. For example, the webkit >>> release bot's results are at >>> http://build.chromium.org/buildbot/layout_test_results/webkit-rel/results.json. >>> That >>> file holds all the historical data for that bot and is copied over during >>> the archive step of each run. We intentionally limit the number of results >>> we keep in that file to 750 runs to keep filesize down. In my accidental >>> checking, I changed 750 to 9. :( >> >> A little bit unrelated: This data, along with all the data on >> build.chromium.org, is replicated on at least 4 machines. It would be >> easy to recover the data if the server dies for example. We are also >> planning to do daily backups, but the data is huge. For example, we archive >> 25GB of new layout test results every day. >> >> Nicolas >> >> >>> A trivial to implement "backup" would be to also copy the file to the >>> archive location for just that run (same place as where we copy >>> layout_test_results.zip), e.g. also copy it to >>> http://build.chromium.org/buildbot/layout_test_results/webkit-rel/29056/. >>> The downside is that this uses up disk space (e.g. the largest results.json >>> file was 25mb before being clobbered). >>> >>> Another problem with backing up is that you'd also have to find a way to >>> restore from backup that didn't lose data from runs that happened since the >>> problem occurred. Merging the two files results.json should be pretty >>> relatively trivial code, but it's all code that someone would need to write >>> and test. >>> >>> While it sucks, I don't think backing up this data is worth the effort. >>> It's a temporary productivity hit for the team, but we get enough new data >>> to make reasonable decisions relatively quickly. Mistakes like this are very >>> rare. It will become even more rare as coding work on the dashboard winds >>> down. >>> >>> Feel free to have at it if you disagree. >>> >>> Creates the results.json file and it's content: >>> trunk/src/webkit/tools/layout_tests/layout_package/json_results_generator.py >>> Copies the results.json file to the right >>> place: >>> trunk/tools/buildbot/scripts/slave/chromium/archive_layout_test_results.py >>> >>> Ojan >>> >>> >>> On Wed, Oct 14, 2009 at 4:24 PM, Jeremy Orlow <jor...@chromium.org>wrote: >>> >>>> I haven't actually gotten anything done on LocalStorage this week >>>> because I've been doing so many small side projects like this.....but if >>>> it's a priority, sure. >>>> >>>> How about a cron job on some machine that ssh's via a cert into whatever >>>> machines the data is stored on, pulls it back, and dumps it into some dir? >>>> When we start filling up the hard drive, we can look at doing something >>>> smarter, deleting old data, or putting it somewhere like GFS. >>>> >>>> What server can I use and where's the data stored? >>>> >>>> On Wed, Oct 14, 2009 at 4:17 PM, Evan Martin <e...@chromium.org> wrote: >>>> >>>>> Sounds like we've got a volunteer! :D :D :D >>>>> >>>>> On Wed, Oct 14, 2009 at 4:15 PM, Jeremy Orlow <jor...@chromium.org> >>>>> wrote: >>>>> > I assume we're going to start backing this data up from now on? >>>>> > >>>>> > On Wed, Oct 14, 2009 at 4:13 PM, Peter Kasting <pkast...@google.com> >>>>> wrote: >>>>> >> >>>>> >> On Wed, Oct 14, 2009 at 3:58 PM, Ojan Vafai <o...@google.com> >>>>> wrote: >>>>> >>> >>>>> >>> I accidentally checked in some test code (one number was wrong!) >>>>> and >>>>> >>> clobbered all but 10 of the runs of data for each builder. There's >>>>> no way to >>>>> >>> recover it. >>>>> >> >>>>> >> Do you moonlight for the Danger team at Microsoft? >>>>> >> PK >>>>> >> >>>>> > >>>>> > >>>>> > > >>>>> > >>>>> >>>> >>>> >>> >>> >>> >> > > > > --~--~---------~--~----~------------~-------~--~----~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~----------~----~----~----~------~----~------~--~---
