How about all of them <https://amplab.cs.berkeley.edu/jenkins/view/Spark/>? How much data per day would it roughly be if we uploaded all the logs for all these builds?
Also, would Databricks be willing to offer up an S3 bucket for this purpose? Nick On Mon Dec 15 2014 at 11:48:44 AM shane knapp <skn...@berkeley.edu> wrote: > right now, the following logs are archived on to the master: > > local log_files=$( > find .\ > -name "unit-tests.log" -o\ > -path "./sql/hive/target/HiveCompatibilitySuite.failed" -o\ > -path "./sql/hive/target/HiveCompatibilitySuite.hiveFailed" -o\ > -path "./sql/hive/target/HiveCompatibilitySuite.wrong" > ) > > regarding dumping stuff to S3 -- thankfully, since we're not looking at a > lot of disk usage, i don't see a problem w/this. we could tar/zip up the > XML for each build and just dump it there. > > what builds are we thinking about? spark pull request builder? what > others? > > On Mon, Dec 15, 2014 at 1:33 AM, Nicholas Chammas < > nicholas.cham...@gmail.com> wrote: >> >> Every time we run a test cycle on our Jenkins cluster, we generate >> hundreds >> of XML reports covering all the tests we have (e.g. >> >> `streaming/target/test-reports/org.apache.spark.streaming.util.WriteAheadLogSuite.xml`). >> >> These reports contain interesting information about whether tests >> succeeded >> or failed, and how long they took to complete. There is also detailed >> information about the environment they ran in. >> >> It might be valuable to have a window into all these reports across all >> Jenkins builds and across all time, and use that to track basic statistics >> about our tests. That could give us basic insight into what tests are >> flaky >> or slow, and perhaps drive other improvements to our testing >> infrastructure >> that we can't see just yet. >> >> Do people think that would be valuable? Do we already have something like >> this? >> >> I'm thinking for starters it might be cool if we automatically uploaded >> all >> the XML test reports from the Master and the Pull Request builders to an >> S3 >> bucket and just opened it up for the dev community to analyze. >> >> Nick >> >
