In terms of matching, it seems like people should be able to express the portion of nodes that they match. If they want to match the file list length but not the order, they should be able to express that. The important part of a framework is to not require people to overly constrain the output as that causes excessive test maintenance.
With regards to the plan, I think it would be good to take another pass and how we make those as user-friendly as possible. For example, use column names rather than ordinal positions in expressions and avoid using toString. However, I think the only tests that should be related to that should be test presentation tests (not planning tests). -- Jacques Nadeau CTO and Co-Founder, Dremio On Tue, Oct 27, 2015 at 10:13 AM, rahul challapalli < challapallira...@gmail.com> wrote: > With the json tree matching, we still want to control what determines an > equality between 2 json tree's at the level of each operator. We should be > able to ignore the attributes/child-nodes which represent any costing > information or moving parts. With this requirement don't we still need to > come up with CustomMatcher? > > Even if we end up using the JSON plan for the majority of the functional > tests, we should still have tests against the text version of the plan. Can > we somehow standardize on the contents of the text plan instead of relying > on the toString method of the respective operators? > > - Rahul > > On Tue, Oct 27, 2015 at 9:25 AM, Jacques Nadeau <jacq...@dremio.com> > wrote: > > > We should use the structured response (and do structural comparisons) for > > plan verification. The text plan should be cleaned up so that it is more > > useful and user friendly. We definitely shouldn't depend on what is > > effectively a random toString method for functional verification (I know > we > > do now). We should figure out a way to implement the abstract > functionality > > verification using a builder pattern or similar. Think Calcite's > RelBuilder > > with a more generic interfaces. Maybe we create a JsonRelMatcher tree and > > then we can use that to verify a plan. > > > > As I started to write up an example expression proposal, I start to > think: > > this is arbitrary json tree matching, any way we can use something that > > already exists. For example, there are probably 100 json libraries that > do > > basically this... > > > > thoughts? > > > > -- > > Jacques Nadeau > > CTO and Co-Founder, Dremio > > > > On Mon, Oct 26, 2015 at 6:32 PM, rahul challapalli < > > challapallira...@gmail.com> wrote: > > > > > Drillers, > > > > > > We have a handful of tests in the drill test framework (Refer to [1] > for > > > some examples) which compare the text version of the plan to an > expected > > > result. The goal is to compare the structure of the text plan and also > > > selected attributes of some operators ( Eg : numFiles, list of scanned > > > files). The current implementation uses a regex based comparison where > > the > > > QA person constructs an equivalent regex based expected file. This > > reduces > > > the QA productivity and is not flexible w.r.t to moving parts in the > > plan. > > > Any thoughts around this? > > > > > > Few suggestions : > > > 1. Run explain plan with a flag so that the text plan returned is > > > deterministic and does not contain any moving parts. This version of > the > > > plan returned can contain the list of files scanned in an > alphabetically > > > sorted order or in some way which is deterministic. > > > 2. Use the JSON plan and do an operator DAG based comparison. This can > be > > > very flexible. Even with this we still need to have some tests against > > the > > > Text version of the plan. > > > > > > [1] > > > > > > > > > https://github.com/mapr/drill-test-framework/tree/master/framework/resources/Functional/partition_pruning/dfs/csv/plan > > > > > > - Rahul > > > > > >
