Hi, +1 to this idea. I think standardizing the format of the presented test run results makes sense. I also agree that we don't necessarily need to enforce it in any hard way. However, given that we have dashboards of these three major providers, we could consider enforcing the presence of *some* dashboard that shows results of regular system tests executions for any new provider. WDYT?
Best, Michal On Sun, Jun 23, 2024 at 10:09 PM Freddy Demiane <fdemi...@google.com.invalid> wrote: > Hello, > > Thank you for the comments! Indeed, +1 to the idea, I believe this would be > a good step to increase the quality of providers. From our (Google) side, > the dashboard's CI outputs the results to a database, which are then used > to generate an HTML page. Yet, generating and publishing a JSON or a JUnit > XML style file would be a simple task for us. The information in our > database is similar to the structure of the AWS providers json file > https://aws-mwaa.github.io/open-source/system-tests/dashboard.json + > a field for logs. We also have an extra field that specifies the commit-id > against which the CI was run, which I believe is helpful in case users want > to know whether their PR was merged before or after a failure. > If we want to go with the junit-xml style format (I checked this reference > < > https://www.ibm.com/docs/en/developer-for-zos/16.0?topic=formats-junit-xml-format > >), > one thing I could think of is to make each "Dashboard CI run" generate an > xml file where each test is represented by a testcase, which as Jarek > mentioned, could be used in some way in the canary builds. > Let me know what you think. > > Best, > Freddy > > > On Fri, Jun 21, 2024 at 11:12 AM Jarek Potiuk <ja...@potiuk.com> wrote: > > > This is a fantastic idea! I love it ! > > > > It also has some very far reaching possible spin-offs in the future - > > literally few days ago, when I discussed some of the future security > > related work that we might want to do, there was a concept of having a > sort > > of CI of all CIs where we (and by we I mean wider Python ecosystem) could > > gather a status of pre-release versions of dependencies before they hit > > release stage, and some kind of interchange between those CI systems that > > will be machine-parseable is pretty much prerequisite for that. So we > could > > generally try it out and sort out some issues, see how it works in our > > small "airflow" world, but in the future we might be able to use similar > > mechanisms to get alerts for a number of our dependencies - and even > > further than that, we could make such approach much more wide-spread (I > am > > discussing it with people from Python Software Foundation/Packaging team > / > > Python security, so there is a chance this might actually materialize in > a > > long term). This would be the first step. > > > > I think the first step for it could be rather simple and we do not have > to > > invent our own standard - we could easily start with junit-xml style > output > > produced by each dashboard and available under some URL that we could > pull > > in our canary builds and have a step in our canary builds that could > > aggregate multiple xmlunit files coming from various dashboards, display > > them as the output, and fail the job in case some tests are failing (with > > maybe some thresholds). Pytest and a number of tools natively supports > the > > junit-xml format, it's pretty established as machine-readable test > results, > > and I think it has all we need to start with > > > > > https://docs.pytest.org/en/latest/how-to/usage.html#creating-junitxml-format-files > > . > > There is a lot of tooling around this format - including easy ways how we > > could possibly integrate it with Github Actions output (think links to > the > > tests that failed directly in GitHub UI), showing logs of failed tests > etc. > > etc. > > > > If we can get the Astronomer, Amazon and Google team on board with it, we > > could likely implement a simple version quickly and iterate over it - > later > > we could think about possibly evolving that into a more extensible > > approach. > > > > J. > > > > > > > > > > On Thu, Jun 20, 2024 at 11:27 PM Ferruzzi, Dennis > > <ferru...@amazon.com.invalid> wrote: > > > > > Congrats to the Google team for getting their dashboard live, it looks > > > great! I've been thinking of something for a while and thought I'd > > mention > > > it here. I'm wearing a few different hats here so I'll try to clarify > > > context on my plural pronouns the best I can. > > > > > > Now that we [Providers] have a couple of big dashboards up, I'm curious > > if > > > we [Airflow dev community] might collaborate on a community "optional > > > guideline" for a json (or yaml or whatever) format output on the > > dashboards > > > for any providers interested in participating. I'm not interested in > (or > > > trying to) impose any kind of hard-line policy or standard here, but I > > > wonder if we [owners of the existing dashboards] might set some > > non-binding > > > precedent for future providers to join. If others don't follow suit, > > then > > > they wouldn't benefit from whatever uses folks come up with for the > data, > > > but I personally don't think we [Airflow] can or should try to impose > > this > > > on providers. > > > > > > To my knowledge there are three provider-owned system test dashboards > > > currently live, and I look forward to seeing more in time: > > > > > > Astronomer (found this LLM-specific one, not sure if there is another > > > one): https://astronomer.github.io/llm-dags-dashboard/ > > > AWS: > https://aws-mwaa.github.io/open-source/system-tests/dashboard.html > > > and > https://aws-mwaa.github.io/open-source/system-tests/dashboard.json > > > Google: > > > https://storage.googleapis.com/providers-dashboard-html/dashboard.html > > > > > > Each was developed independently, and the path/name of the Google one > may > > > hint that there is already an alternative to the html view that I'm > just > > > not familiar with, so maybe we [the three providers] could collaborate > on > > > some precedent that others could follow? We [AWS] already have ours > > > exporting in json so discussion might start there and see where it > goes? > > > Either way... Even if we [Airflow] don't do anything with the json, I > > bet a > > > user could find interesting things to build if we give them the tools. > > > Maybe aggregating a dashboard which monitors (and alerts?) the status > of > > > the system tests which cover the operators their workflow depends on, > > > maybe? Who knows what someone may come up with once they have the > tools > > to > > > mix and match the data from various providers. > > > > > > Is there any interest in the idea of a "standard json schema" for these > > > and any future system test dashboards? > > > > > > > > > - ferruzzi > > > > > >
