With that in mind it seems the somewhat recurring discussion on coming up with a language independent standard for logical query plans (https://lists.apache.org/thread.html/rfab15e09c97a8fb961d6c5db8b2093824c58d11a51981a40f40cc2c0%40%3Cdev.arrow.apache.org%3E) would be relevant. Each test case would then be a triplet of (Input Dataframe, Logical Plan, Output Dataframe). So perhaps tackling this effort would be to make progress on both fronts.
On Fri, May 14, 2021 at 7:39 AM Julian Hyde <jhyde.apa...@gmail.com> wrote: > > Do these any of these compute functions have analogs in other implementations > of Arrow (e.g. Rust)? > > I believe that as much as possible of Arrow’s compute functionality should be > cross-language. Perhaps there are language-specific differences in how > functions are invoked, but the basic functionality is the same. > > If people buy into that premise, then a single suite of tests is a powerful > way to make that happen. The tests can be written in a high-level language > and can generate tests in each implementation language. (For these purposes, > the “high-level language” could be a special text format, could be a data > language such as JSON, or could be a programming language such as Python; it > doesn’t matter much.) > > For example, > > assertThatCall(“foo(1, 2)”, returns(“3”)) > > might actually call foo with arguments 1 and 2, or it might generate a C++ or > Rust test that does the same. > > > Julian > > > > On May 14, 2021, at 8:45 AM, Antoine Pitrou <anto...@python.org> wrote: > > > > > > Le 14/05/2021 à 15:30, Wes McKinney a écrit : > >> hi folks, > >> As we build more functions (kernels) in the project, I note that the > >> amount of hand-coded C++ code relating to testing function correctness > >> is growing significantly. Many of these tests are quite simple and > >> could be expressed in a text format that can be parsed and evaluated. > >> Thoughts about building something like that to make it easier to write > >> functional correctness tests? > > > > Or perhaps build-up higher level C++ functions if desired? > > > > Or even write some of those tests as part of the PyArrow test suite. > > > > I'm not sure adding a custom (and probably inflexible) text format is > > really a good use of our time. > > > > Regards > > > > Antoine. >
