This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new eb6ebc8f5 Publish built docs triggered by
8a1a154f7b60e5821cb60cd1c312e645ad3aeb4f
eb6ebc8f5 is described below
commit eb6ebc8f5b367e55d7359ce6a01b3c4ff93bf129
Author: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
AuthorDate: Thu Mar 6 01:48:43 2025 +0000
Publish built docs triggered by 8a1a154f7b60e5821cb60cd1c312e645ad3aeb4f
---
_sources/contributor-guide/development.md.txt | 21 +++++++++++++++++++--
contributor-guide/development.html | 25 +++++++++++++++++++------
searchindex.js | 2 +-
3 files changed, 39 insertions(+), 9 deletions(-)
diff --git a/_sources/contributor-guide/development.md.txt
b/_sources/contributor-guide/development.md.txt
index 737c0aaf5..a9c43103b 100644
--- a/_sources/contributor-guide/development.md.txt
+++ b/_sources/contributor-guide/development.md.txt
@@ -92,9 +92,23 @@ Other options for selecting specific suites are described in
the [ScalaTest Mave
## Plan Stability Testing
Comet has a plan stability testing framework that can be used to test the
stability of the query plans generated by Comet.
-The plan stability testing framework is located in the `spark` module and can
be run using the following command:
+The plan stability testing framework is located in the `spark` module and can
be run using the following commands.
+
+First, Comet needs to be installed for each Spark version to be tested:
+
+```sh
+./mvnw install -DskipTests -Pspark-3.4
+./mvnw install -DskipTests -Pspark-3.5
+# note that Spark 4.0 requires JDK 17 or later
+./mvnw install -DskipTests -Pspark-4.0
+```
+
+Note that the output files get written to `$SPARK_HOME`.
+
+The tests can be run with:
```sh
+export SPARK_HOME=`pwd`
./mvnw -pl spark
-Dsuites="org.apache.spark.sql.comet.CometTPCDSV1_4_PlanStabilitySuite" -nsu
test
./mvnw -pl spark
-Dsuites="org.apache.spark.sql.comet.CometTPCDSV1_4_PlanStabilitySuite"
-Pspark-3.5 -nsu test
./mvnw -pl spark
-Dsuites="org.apache.spark.sql.comet.CometTPCDSV1_4_PlanStabilitySuite"
-Pspark-4.0 -nsu test
@@ -102,15 +116,17 @@ The plan stability testing framework is located in the
`spark` module and can be
and
```sh
+export SPARK_HOME=`pwd`
./mvnw -pl spark
-Dsuites="org.apache.spark.sql.comet.CometTPCDSV2_7_PlanStabilitySuite" -nsu
test
./mvnw -pl spark
-Dsuites="org.apache.spark.sql.comet.CometTPCDSV2_7_PlanStabilitySuite"
-Pspark-3.5 -nsu test
./mvnw -pl spark
-Dsuites="org.apache.spark.sql.comet.CometTPCDSV2_7_PlanStabilitySuite"
-Pspark-4.0 -nsu test
```
If your pull request changes the query plans generated by Comet, you should
regenerate the golden files.
-To regenerate the golden files, you can run the following command:
+To regenerate the golden files, you can run the following commands.
```sh
+export SPARK_HOME=`pwd`
SPARK_GENERATE_GOLDEN_FILES=1 ./mvnw -pl spark
-Dsuites="org.apache.spark.sql.comet.CometTPCDSV1_4_PlanStabilitySuite" -nsu
test
SPARK_GENERATE_GOLDEN_FILES=1 ./mvnw -pl spark
-Dsuites="org.apache.spark.sql.comet.CometTPCDSV1_4_PlanStabilitySuite"
-Pspark-3.5 -nsu test
SPARK_GENERATE_GOLDEN_FILES=1 ./mvnw -pl spark
-Dsuites="org.apache.spark.sql.comet.CometTPCDSV1_4_PlanStabilitySuite"
-Pspark-4.0 -nsu test
@@ -118,6 +134,7 @@ SPARK_GENERATE_GOLDEN_FILES=1 ./mvnw -pl spark
-Dsuites="org.apache.spark.sql.co
and
```sh
+export SPARK_HOME=`pwd`
SPARK_GENERATE_GOLDEN_FILES=1 ./mvnw -pl spark
-Dsuites="org.apache.spark.sql.comet.CometTPCDSV2_7_PlanStabilitySuite" -nsu
test
SPARK_GENERATE_GOLDEN_FILES=1 ./mvnw -pl spark
-Dsuites="org.apache.spark.sql.comet.CometTPCDSV2_7_PlanStabilitySuite"
-Pspark-3.5 -nsu test
SPARK_GENERATE_GOLDEN_FILES=1 ./mvnw -pl spark
-Dsuites="org.apache.spark.sql.comet.CometTPCDSV2_7_PlanStabilitySuite"
-Pspark-4.0 -nsu test
diff --git a/contributor-guide/development.html
b/contributor-guide/development.html
index c0b1d44c1..d32dcad7c 100644
--- a/contributor-guide/development.html
+++ b/contributor-guide/development.html
@@ -468,27 +468,40 @@ in their name in <code class="docutils literal
notranslate"><span class="pre">or
<section id="plan-stability-testing">
<h2>Plan Stability Testing<a class="headerlink" href="#plan-stability-testing"
title="Link to this heading">ΒΆ</a></h2>
<p>Comet has a plan stability testing framework that can be used to test the
stability of the query plans generated by Comet.
-The plan stability testing framework is located in the <code class="docutils
literal notranslate"><span class="pre">spark</span></code> module and can be
run using the following command:</p>
-<div class="highlight-sh notranslate"><div
class="highlight"><pre><span></span>./mvnw<span class="w"> </span>-pl<span
class="w"> </span>spark<span class="w"> </span>-Dsuites<span
class="o">=</span><span
class="s2">"org.apache.spark.sql.comet.CometTPCDSV1_4_PlanStabilitySuite"</span><span
class="w"> </span>-nsu<span class="w"> </span><span class="nb">test</span>
+The plan stability testing framework is located in the <code class="docutils
literal notranslate"><span class="pre">spark</span></code> module and can be
run using the following commands.</p>
+<p>First, Comet needs to be installed for each Spark version to be tested:</p>
+<div class="highlight-sh notranslate"><div
class="highlight"><pre><span></span>./mvnw<span class="w"> </span>install<span
class="w"> </span>-DskipTests<span class="w"> </span>-Pspark-3.4
+./mvnw<span class="w"> </span>install<span class="w"> </span>-DskipTests<span
class="w"> </span>-Pspark-3.5<span class="w"> </span>
+<span class="c1"># note that Spark 4.0 requires JDK 17 or later</span>
+./mvnw<span class="w"> </span>install<span class="w"> </span>-DskipTests<span
class="w"> </span>-Pspark-4.0
+</pre></div>
+</div>
+<p>Note that the output files get written to <code class="docutils literal
notranslate"><span class="pre">$SPARK_HOME</span></code>.</p>
+<p>The tests can be run with:</p>
+<div class="highlight-sh notranslate"><div
class="highlight"><pre><span></span><span class="nb">export</span><span
class="w"> </span><span class="nv">SPARK_HOME</span><span
class="o">=</span><span class="sb">`</span><span class="nb">pwd</span><span
class="sb">`</span>
+./mvnw<span class="w"> </span>-pl<span class="w"> </span>spark<span class="w">
</span>-Dsuites<span class="o">=</span><span
class="s2">"org.apache.spark.sql.comet.CometTPCDSV1_4_PlanStabilitySuite"</span><span
class="w"> </span>-nsu<span class="w"> </span><span class="nb">test</span>
./mvnw<span class="w"> </span>-pl<span class="w"> </span>spark<span class="w">
</span>-Dsuites<span class="o">=</span><span
class="s2">"org.apache.spark.sql.comet.CometTPCDSV1_4_PlanStabilitySuite"</span><span
class="w"> </span>-Pspark-3.5<span class="w"> </span>-nsu<span class="w">
</span><span class="nb">test</span>
./mvnw<span class="w"> </span>-pl<span class="w"> </span>spark<span class="w">
</span>-Dsuites<span class="o">=</span><span
class="s2">"org.apache.spark.sql.comet.CometTPCDSV1_4_PlanStabilitySuite"</span><span
class="w"> </span>-Pspark-4.0<span class="w"> </span>-nsu<span class="w">
</span><span class="nb">test</span>
</pre></div>
</div>
<p>and</p>
-<div class="highlight-sh notranslate"><div
class="highlight"><pre><span></span>./mvnw<span class="w"> </span>-pl<span
class="w"> </span>spark<span class="w"> </span>-Dsuites<span
class="o">=</span><span
class="s2">"org.apache.spark.sql.comet.CometTPCDSV2_7_PlanStabilitySuite"</span><span
class="w"> </span>-nsu<span class="w"> </span><span class="nb">test</span>
+<div class="highlight-sh notranslate"><div
class="highlight"><pre><span></span><span class="nb">export</span><span
class="w"> </span><span class="nv">SPARK_HOME</span><span
class="o">=</span><span class="sb">`</span><span class="nb">pwd</span><span
class="sb">`</span>
+./mvnw<span class="w"> </span>-pl<span class="w"> </span>spark<span class="w">
</span>-Dsuites<span class="o">=</span><span
class="s2">"org.apache.spark.sql.comet.CometTPCDSV2_7_PlanStabilitySuite"</span><span
class="w"> </span>-nsu<span class="w"> </span><span class="nb">test</span>
./mvnw<span class="w"> </span>-pl<span class="w"> </span>spark<span class="w">
</span>-Dsuites<span class="o">=</span><span
class="s2">"org.apache.spark.sql.comet.CometTPCDSV2_7_PlanStabilitySuite"</span><span
class="w"> </span>-Pspark-3.5<span class="w"> </span>-nsu<span class="w">
</span><span class="nb">test</span>
./mvnw<span class="w"> </span>-pl<span class="w"> </span>spark<span class="w">
</span>-Dsuites<span class="o">=</span><span
class="s2">"org.apache.spark.sql.comet.CometTPCDSV2_7_PlanStabilitySuite"</span><span
class="w"> </span>-Pspark-4.0<span class="w"> </span>-nsu<span class="w">
</span><span class="nb">test</span>
</pre></div>
</div>
<p>If your pull request changes the query plans generated by Comet, you should
regenerate the golden files.
-To regenerate the golden files, you can run the following command:</p>
-<div class="highlight-sh notranslate"><div
class="highlight"><pre><span></span><span
class="nv">SPARK_GENERATE_GOLDEN_FILES</span><span class="o">=</span><span
class="m">1</span><span class="w"> </span>./mvnw<span class="w">
</span>-pl<span class="w"> </span>spark<span class="w"> </span>-Dsuites<span
class="o">=</span><span
class="s2">"org.apache.spark.sql.comet.CometTPCDSV1_4_PlanStabilitySuite"</span><span
class="w"> </span>-nsu<span class="w"> </span><span class="nb">test</span>
+To regenerate the golden files, you can run the following commands.</p>
+<div class="highlight-sh notranslate"><div
class="highlight"><pre><span></span><span class="nb">export</span><span
class="w"> </span><span class="nv">SPARK_HOME</span><span
class="o">=</span><span class="sb">`</span><span class="nb">pwd</span><span
class="sb">`</span>
+<span class="nv">SPARK_GENERATE_GOLDEN_FILES</span><span
class="o">=</span><span class="m">1</span><span class="w"> </span>./mvnw<span
class="w"> </span>-pl<span class="w"> </span>spark<span class="w">
</span>-Dsuites<span class="o">=</span><span
class="s2">"org.apache.spark.sql.comet.CometTPCDSV1_4_PlanStabilitySuite"</span><span
class="w"> </span>-nsu<span class="w"> </span><span class="nb">test</span>
<span class="nv">SPARK_GENERATE_GOLDEN_FILES</span><span
class="o">=</span><span class="m">1</span><span class="w"> </span>./mvnw<span
class="w"> </span>-pl<span class="w"> </span>spark<span class="w">
</span>-Dsuites<span class="o">=</span><span
class="s2">"org.apache.spark.sql.comet.CometTPCDSV1_4_PlanStabilitySuite"</span><span
class="w"> </span>-Pspark-3.5<span class="w"> </span>-nsu<span class="w">
</span><span class="nb">test</span>
<span class="nv">SPARK_GENERATE_GOLDEN_FILES</span><span
class="o">=</span><span class="m">1</span><span class="w"> </span>./mvnw<span
class="w"> </span>-pl<span class="w"> </span>spark<span class="w">
</span>-Dsuites<span class="o">=</span><span
class="s2">"org.apache.spark.sql.comet.CometTPCDSV1_4_PlanStabilitySuite"</span><span
class="w"> </span>-Pspark-4.0<span class="w"> </span>-nsu<span class="w">
</span><span class="nb">test</span>
</pre></div>
</div>
<p>and</p>
-<div class="highlight-sh notranslate"><div
class="highlight"><pre><span></span><span
class="nv">SPARK_GENERATE_GOLDEN_FILES</span><span class="o">=</span><span
class="m">1</span><span class="w"> </span>./mvnw<span class="w">
</span>-pl<span class="w"> </span>spark<span class="w"> </span>-Dsuites<span
class="o">=</span><span
class="s2">"org.apache.spark.sql.comet.CometTPCDSV2_7_PlanStabilitySuite"</span><span
class="w"> </span>-nsu<span class="w"> </span><span class="nb">test</span>
+<div class="highlight-sh notranslate"><div
class="highlight"><pre><span></span><span class="nb">export</span><span
class="w"> </span><span class="nv">SPARK_HOME</span><span
class="o">=</span><span class="sb">`</span><span class="nb">pwd</span><span
class="sb">`</span>
+<span class="nv">SPARK_GENERATE_GOLDEN_FILES</span><span
class="o">=</span><span class="m">1</span><span class="w"> </span>./mvnw<span
class="w"> </span>-pl<span class="w"> </span>spark<span class="w">
</span>-Dsuites<span class="o">=</span><span
class="s2">"org.apache.spark.sql.comet.CometTPCDSV2_7_PlanStabilitySuite"</span><span
class="w"> </span>-nsu<span class="w"> </span><span class="nb">test</span>
<span class="nv">SPARK_GENERATE_GOLDEN_FILES</span><span
class="o">=</span><span class="m">1</span><span class="w"> </span>./mvnw<span
class="w"> </span>-pl<span class="w"> </span>spark<span class="w">
</span>-Dsuites<span class="o">=</span><span
class="s2">"org.apache.spark.sql.comet.CometTPCDSV2_7_PlanStabilitySuite"</span><span
class="w"> </span>-Pspark-3.5<span class="w"> </span>-nsu<span class="w">
</span><span class="nb">test</span>
<span class="nv">SPARK_GENERATE_GOLDEN_FILES</span><span
class="o">=</span><span class="m">1</span><span class="w"> </span>./mvnw<span
class="w"> </span>-pl<span class="w"> </span>spark<span class="w">
</span>-Dsuites<span class="o">=</span><span
class="s2">"org.apache.spark.sql.comet.CometTPCDSV2_7_PlanStabilitySuite"</span><span
class="w"> </span>-Pspark-4.0<span class="w"> </span>-nsu<span class="w">
</span><span class="nb">test</span>
</pre></div>
diff --git a/searchindex.js b/searchindex.js
index ca881f4ef..25d8aa729 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"1. Install Comet": [[9, "install-comet"]], "2.
Clone Spark and Apply Diff": [[9, "clone-spark-and-apply-diff"]], "3. Run Spark
SQL Tests": [[9, "run-spark-sql-tests"]], "ANSI mode": [[11, "ansi-mode"]],
"API Differences Between Spark Versions": [[0,
"api-differences-between-spark-versions"]], "ASF Links": [[10, null]], "Adding
Spark-side Tests for the New Expression": [[0,
"adding-spark-side-tests-for-the-new-expression"]], "Adding a New Expression":
[[0, [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"1. Install Comet": [[9, "install-comet"]], "2.
Clone Spark and Apply Diff": [[9, "clone-spark-and-apply-diff"]], "3. Run Spark
SQL Tests": [[9, "run-spark-sql-tests"]], "ANSI mode": [[11, "ansi-mode"]],
"API Differences Between Spark Versions": [[0,
"api-differences-between-spark-versions"]], "ASF Links": [[10, null]], "Adding
Spark-side Tests for the New Expression": [[0,
"adding-spark-side-tests-for-the-new-expression"]], "Adding a New Expression":
[[0, [...]
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
