hi Areg, This sounds great -- we've discussed building a more full-featured benchmark automation system in the past but nothing has been developed yet.
Your proposal about the details sounds OK; the single most important thing to me is that we build and maintain a very general purpose database schema for building the historical benchmark database The benchmark database should keep track of: * Timestamp of benchmark run * Git commit hash of codebase * Machine unique name (sort of the "user id") * CPU identification for machine, and clock frequency (in case of overclocking) * CPU cache sizes (L1/L2/L3) * Whether or not CPU throttling is enabled (if it can be easily determined) * RAM size * GPU identification (if any) * Benchmark unique name * Programming language(s) associated with benchmark (e.g. a benchmark may involve both C++ and Python) * Benchmark time, plus mean and standard deviation if available, else NULL (maybe some other things) I would rather not be locked into the internal database schema of a particular benchmarking tool. So people in the community can just run SQL queries against the database and use the data however they like. We'll just have to be careful that people don't DROP TABLE or DELETE (but we should have daily backups so we can recover from such cases) So while we may make use of TeamCity to schedule the runs on the cloud and physical hardware, we should also provide a path for other people in the community to add data to the benchmark database on their hardware on an ad hoc basis. For example, I have several machines in my home on all operating systems (Windows / macOS / Linux, and soon also ARM64) and I'd like to set up scheduled tasks / cron jobs to report in to the database at least on a daily basis. Ideally the benchmark database would just be a PostgreSQL server with a schema we write down and keep backed up etc. Hosted PostgreSQL is inexpensive ($200+ per year depending on size of instance; this probably doesn't need to be a crazy big machine) I suspect there will be a manageable amount of development involved to glue each of the benchmarking frameworks together with the benchmark database. This can also handle querying the operating system for the system information listed above Thanks Wes On Fri, Jan 18, 2019 at 12:14 AM Melik-Adamyan, Areg <areg.melik-adam...@intel.com> wrote: > > Hello, > > I want to restart/attach to the discussions for creating Arrow benchmarking > dashboard. I want to propose performance benchmark run per commit to track > the changes. > The proposal includes building infrastructure for per-commit tracking > comprising of the following parts: > - Hosted JetBrains for OSS https://teamcity.jetbrains.com/ as a build system > - Agents running in cloud both VM/container (DigitalOcean, or others) and > bare-metal (Packet.net/AWS) and on-premise(Nvidia boxes?) > - JFrog artifactory storage and management for OSS projects > https://jfrog.com/open-source/#artifactory2 > - Codespeed as a frontend https://github.com/tobami/codespeed > > I am volunteering to build such system (if needed more Intel folks will be > involved) so we can start tracking performance on various platforms and > understand how changes affect it. > > Please, let me know your thoughts! > > Thanks, > -Areg. > > >
