andygrove commented on code in PR #316:
URL: https://github.com/apache/datafusion-spark/pull/316#discussion_r4615381061


##########
crates/sqllogictest/tests/sqllogictests.rs:
##########
@@ -83,21 +242,44 @@ async fn main() -> ExitCode {
         {
             continue;
         }
-        handles.push((relative.clone(), tokio::spawn(run_file(path, 
relative))));
+        for &mode in &modes {
+            let task = tokio::spawn(run_file(path.clone(), relative.clone(), 
mode));
+            handles.push((relative.clone(), mode, task));
+        }
     }
 
     let total = handles.len();
     let mut failed = 0;
-    for (relative, handle) in handles {
-        match handle.await.expect("test task panicked") {
-            Ok(()) => println!("PASS {}", relative.display()),
-            Err(e) => {
+    for (relative, mode, handle) in handles {
+        let name = format!("[{}] {}", mode.label(), relative.display());
+        let known_failure = EVALUATE_KNOWN_FAILURES
+            .iter()
+            .find(|(file, _)| mode == Mode::Evaluate && Path::new(file) == 
relative)
+            .map(|(_, issue)| *issue);
+        // A panic in one file (for example a function indexing past the end 
of an
+        // array) is reported as that file failing, so the other results still 
print.
+        match (
+            handle.await.map_err(|e| e.to_string()).and_then(|r| r),
+            known_failure,
+        ) {
+            (Ok(()), None) => println!("PASS {name}"),
+            (Err(_), Some(issue)) => {

Review Comment:
   Done in 59ed9f9. EVALUATE_KNOWN_FAILURES is gone. Each record that calls a 
function with no invoke_with_args yet is marked `skipif evaluate`, and a note 
at the top of the file links the issue. The rest of each file now runs in 
evaluate mode. I checked base64.slt: the unbase64 records run, and changing one 
expected value fails the run.



##########
crates/sqllogictest/test_files/README.md:
##########
@@ -69,6 +69,27 @@ SELECT expm1(0::INT);
 SELECT expm1(a) FROM (VALUES (0::INT), (1::INT)) AS t(a);
 ```
 
+## Simplify and evaluate modes
+
+`cargo test -p datafusion-spark-sqllogictest` runs every file twice:
+
+- `simplify` uses the default optimizer, so a call whose arguments are all 
literals,
+  such as `SELECT expm1(0::INT)`, is constant-folded while planning. This 
calls the
+  function's `simplify` (if it has one) and evaluates it on a one-row batch.
+- `evaluate` removes the `simplify_expressions` optimizer rule, so every call 
runs

Review Comment:
   Narrowed in 59ed9f9. The README now says that evaluate mode only changes the 
optimizer, that SQL planning can still evaluate some expressions first, such as 
table function arguments, and that a call over rows is the reliable way to test 
invoke_with_args.



##########
crates/sqllogictest/tests/sqllogictests.rs:
##########
@@ -83,21 +242,44 @@ async fn main() -> ExitCode {
         {
             continue;
         }
-        handles.push((relative.clone(), tokio::spawn(run_file(path, 
relative))));
+        for &mode in &modes {
+            let task = tokio::spawn(run_file(path.clone(), relative.clone(), 
mode));
+            handles.push((relative.clone(), mode, task));
+        }
     }
 
     let total = handles.len();
     let mut failed = 0;
-    for (relative, handle) in handles {
-        match handle.await.expect("test task panicked") {
-            Ok(()) => println!("PASS {}", relative.display()),
-            Err(e) => {
+    for (relative, mode, handle) in handles {
+        let name = format!("[{}] {}", mode.label(), relative.display());
+        let known_failure = EVALUATE_KNOWN_FAILURES
+            .iter()
+            .find(|(file, _)| mode == Mode::Evaluate && Path::new(file) == 
relative)
+            .map(|(_, issue)| *issue);
+        // A panic in one file (for example a function indexing past the end 
of an
+        // array) is reported as that file failing, so the other results still 
print.
+        match (
+            handle.await.map_err(|e| e.to_string()).and_then(|r| r),
+            known_failure,
+        ) {
+            (Ok(()), None) => println!("PASS {name}"),
+            (Err(_), Some(issue)) => {
+                println!("XFAIL {name} (known failure, see {issue})")
+            }
+            (Ok(()), Some(issue)) => {
+                failed += 1;
+                println!(
+                    "FAIL {name}\nexpected to fail ({issue}) but passed, \
+                     remove it from EVALUATE_KNOWN_FAILURES\n"
+                );
+            }
+            (Err(e), None) => {
                 failed += 1;
-                println!("FAIL {}\n{e}\n", relative.display());
+                println!("FAIL {name}\n{e}\n");
             }
         }
     }
-    println!("{} of {total} test files passed", total - failed);
+    println!("{} of {total} test runs passed", total - failed);

Review Comment:
   With the XFAIL handling removed in 59ed9f9, every run is a plain pass or 
fail.



##########
crates/sqllogictest/test_files/math/bround.slt:
##########
@@ -145,12 +148,18 @@ SELECT bround(column1, 1) FROM VALUES 
(0.25::decimal(3,2)), (0.35::decimal(3,2))
 NULL
 
 # A constant scale that is not a bare literal
+# The scale is a constant expression, which is a literal only after constant 
folding,
+# so the decimal result type is computed from it only when folding runs.
+skipif evaluate

Review Comment:
   Documented in 59ed9f9 rather than tracked. The planner computes the decimal 
result type from literal arguments only, so a scale like `1::bigint` is not 
seen without constant folding. invoke_with_args is fine. A caller that builds 
the physical expression, such as Comet, passes the result type itself. The 
comment on each skipped record now says this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to