Hi folks, I'm all for keeping things simple. I agree that we should refactor the regtests to some extent. For example, we've discussed enabling real AWS S3 tests for quite a while. Maybe JB and I can help restart that effort.
I looked at a recent CI run from PR #4535. Based on that example, moving regtests to integration tests would not necessarily save time. In fact, it could potentially increase overall CI duration, since the longest running workflows are currently not the regtests. CI/PR / CI/PR / Regression Tests (minio) (pull_request)Successful in 20m CI/PR / CI/PR / Regression Tests (rustfs) (pull_request)Successful in 21m CI/PR / CI/PR / Runtime Service Integration Tests (pull_request)Successful in 24m CI/PR / CI/PR / Other Integration Tests (pull_request)Successful in 25m Given that, I think it would be helpful to first quantify the expected CI savings before we invest significant effort in migrating the existing regtests. Yufei On Mon, Jun 1, 2026 at 2:51 PM Dmitri Bourlatchkov <[email protected]> wrote: > Hi All, > > Reusing the standard Polaris server images produced during CI for > integration tests sound reasonable. I hope these tests will not cause too > much extra CI overhead. > > However, testing various _clients_ is probably not worth the extra CI time. > Polaris presents a REST API to clients. If a particular client is known to > expect a particular behaviour from the server, such an expectation can be > coded in java, under the JUnit framework. > > If we do not know what specific behaviours current regtests are expected to > validate, that alone is probably not a strong argument for keeping them. If > we cannot accept the risk of altering existing regtests, they become a > maintenance burden. > > How about moving the existing regtests to a nightly/weekly schedule based > on `main` and gradually migrating their "logic" to Gradle/CI tests? If > regressions are detected, we'll enhance those new tests to cover the same > expectations. WDYT? > > Thanks. > Dmitri. > > > > On Mon, Jun 1, 2026 at 9:15 AM Robert Stupp <[email protected]> wrote: > > > Hi all, > > > > Currently, the regtests can consume 40+ minutes for PR CI runs, plus CI > for > > the main and release branches. > > That's a significant cost, so I think we should have a good purpose for > > running those every time CI is triggered. > > > > The Spark plugin regtests have, at least to what I can see, coverage that > > looks narrow enough that it could mostly be implemented as JUnit/Gradle > > integration tests. > > > > On the top-level regtests: > > The cloud-provider specific tests do not appear to run, because we do not > > have credentials to run those. > > I could not find any tests that run against another query engine than > Spark > > or PySpark. > > > > I could imagine reducing Docker usage to a minimum, and, for example, use > > the Polaris container image that is already build in CI. > > This can also be used for the Python CLI, if there is no better > > alternative. > > > > I am not saying that end-2-end tests are not useful. > > But I would like to keep the scope of these regtests to be small, > > purposeful, and isolated from ordinary CI cost, unless they protect a > > distinct integration boundary. > > Then we could add other query engines, which are not yet covered, as > well. > > If we want broader engine coverage, such as Trino or DuckDB, that should > be > > explicit and targeted rather than a reason to keep duplicating the > current > > Spark SQL regtest stack. > > > > Robert > > > > > > On Sun, May 31, 2026 at 1:33 AM Michael Collado <[email protected]> > > wrote: > > > > > There are a number of times the e2e purest have caught issues the have > > > tests don’t cover. Personally, I’d love to see expansion of the tests > to > > > cover frameworks that we simply can’t cover in the Java tests (pyspark, > > > duckdb, trino, etc.). The variance in support across engines and > catalogs > > > was kind of a theme at the Iceberg Summit in April. Broader coverage of > > > testing in the ecosystem would really help Polaris in that regard. > > > > > > Mike > > > > > > On Thu, May 28, 2026 at 7:59 PM Dmitri Bourlatchkov <[email protected]> > > > wrote: > > > > > > > Hi Yong, > > > > > > > > Good questions :) As for me I'd refactor all shell-based regtests > that > > > use > > > > Docker into Gradle-based tests (preferably JUnit). > > > > > > > > If people find value in the section of the regtests that talk to real > > > > cloud services (but do not run in CI), we can keep them. Still, we > have > > > > examples of JUnit-based tests that can talk to real cloud services > too. > > > > > > > > However, this is just my opinion. Let's wait a bit for other people > to > > > > express their opinions. We had a good discussion during the sync > call, > > > but > > > > I'm not sure we have consensus yet... hence this thread. > > > > > > > > Cheers, > > > > Dmitri. > > > > > > > > On Thu, May 28, 2026 at 10:00 PM Yong Zheng <[email protected]> > wrote: > > > > > > > > > One clarification, the regtests we talk about here is only > > > > > > > > > https://github.com/apache/polaris/tree/main/plugins/spark/v3.5/regtests > > > > > or https://github.com/apache/polaris/tree/main/regtests as well? > as > > > the > > > > > v3.5/regtests doesn't have cloud dependencies but the later one > has. > > > > > > > > > > Thanks, > > > > > Yong Zheng > > > > > > > > > > On 2026/05/29 01:29:24 Yong Zheng wrote: > > > > > > Hello Dmitri, > > > > > > > > > > > > Thanks for the recap. If I understand correctly, we want to merge > > > test > > > > > coverage from `/regtests` (docker based) into `/integration`? > > Looking > > > at > > > > > them, integration has most of the tests we have in regtests excepts > > the > > > > > ones that are real cloud specific such as spark_sql_gcp* and > > > > > spark_sql_azure* as well as pyspark specific (t_pyspark). The > cloud > > > > > specific ones are not part of CI, in that case, do we still want to > > > > covert > > > > > them? Also, I am not sure if we can convert the pyspark code into > > JUnit > > > > > directly. > > > > > > > > > > > > Also, regarding those duplicates classes, should we move them to > > > > common? > > > > > For the ones that are very similar but minor diff, should we > proceed > > > with > > > > > adapters for version specific (meaning, we won't be follow what > > Iceberg > > > > is > > > > > doing with different version of spark). > > > > > > > > > > > > Lastly, should we close current PR and handle the two above in a > > > > > separate PRs first before revisiting Spark4 support? > > > > > > > > > > > > Thanks, > > > > > > Yong Zheng > > > > > > > > > > > > On 2026/05/28 21:07:44 Dmitri Bourlatchkov wrote: > > > > > > > Hi All, > > > > > > > > > > > > > > This is to recap and follow up on today's Community Sync > > > discussion. > > > > > This > > > > > > > matter came up during the review of the Spark 4 PR, but I > believe > > > it > > > > > > > deserves a dedicated discussion. > > > > > > > > > > > > > > Observations: > > > > > > > > > > > > > > * Tests under the /regtests directory take a considerable > amount > > of > > > > CI > > > > > > > resources. This is mostly due to building custom Docker images. > > > > > > > > > > > > > > * When adding support for newer Spark versions, developers > > > naturally > > > > > tend > > > > > > > to copy existing test infrastructure, which results in building > > > more > > > > > Docker > > > > > > > images. > > > > > > > > > > > > > > * The coverage these tests provide probably does not require > the > > > > heavy > > > > > > > docker machinery. These tests validate that the Spark Session > can > > > > > interact > > > > > > > with Polaris over the Iceberg REST Catalog Java client. The > same > > > > > coverage > > > > > > > can be provided in an isolated JVM running under Gradle more > > > > > efficiently > > > > > > > (without a docker env.) > > > > > > > > > > > > > > Proposal: > > > > > > > > > > > > > > * Gradually convert Spark tests under /regtest to Gradle tasks > > and > > > > > JUnit > > > > > > > tests without Docker. > > > > > > > > > > > > > > The exact impl. is to be figured out. > > > > > > > > > > > > > > Most of the tests can run under the Junit framework using a > local > > > > > Polaris > > > > > > > server (same JVM or different JVM depending on use case). > > > > > > > > > > > > > > True "integration" tests can still run under Gradle as a task > > that > > > > > starts a > > > > > > > Spark shell in a fresh JVM and executes a small set of SQL > > commands > > > > in > > > > > it. > > > > > > > However, these "big" tests can probably be limited in number > and > > > > > complexity > > > > > > > and as such should not generate excessive CI load. More > > > specifically, > > > > > these > > > > > > > tests probably do not need to assert the verbatim output of the > > SQL > > > > > > > commands. A basic sanity check should be sufficient. Functional > > > tests > > > > > can > > > > > > > be performed under JUnit, I think. > > > > > > > > > > > > > > Thoughts? > > > > > > > > > > > > > > Thanks, > > > > > > > Dmitri. > > > > > > > > > > > > > > > > > > > > > > > > > > > >
