This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new e9d970347 Publish built docs triggered by
8f14c72624b678ec9c7843877fe63ccd5ac80183
e9d970347 is described below
commit e9d9703474f78688a13ec46e168727ca49b45e35
Author: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
AuthorDate: Wed May 21 11:23:54 2025 +0000
Publish built docs triggered by 8f14c72624b678ec9c7843877fe63ccd5ac80183
---
_images/img.png | Bin 0 -> 194412 bytes
_sources/contributor-guide/spark-sql-tests.md.txt | 10 +++++++-
contributor-guide/spark-sql-tests.html | 27 ++++++++++++++++++----
searchindex.js | 2 +-
4 files changed, 33 insertions(+), 6 deletions(-)
diff --git a/_images/img.png b/_images/img.png
new file mode 100644
index 000000000..e9d460eed
Binary files /dev/null and b/_images/img.png differ
diff --git a/_sources/contributor-guide/spark-sql-tests.md.txt
b/_sources/contributor-guide/spark-sql-tests.md.txt
index 56e1262ad..2c74f36f1 100644
--- a/_sources/contributor-guide/spark-sql-tests.md.txt
+++ b/_sources/contributor-guide/spark-sql-tests.md.txt
@@ -65,7 +65,7 @@ ENABLE_COMET=true build/sbt "hive/testOnly * -- -l
org.apache.spark.tags.Extende
ENABLE_COMET=true build/sbt "hive/testOnly * -- -n
org.apache.spark.tags.ExtendedHiveTest"
ENABLE_COMET=true build/sbt "hive/testOnly * -- -n
org.apache.spark.tags.SlowHiveTest"
```
-#### Steps to run individual test suites
+#### Steps to run individual test suites through SBT
1. Open SBT with Comet enabled
```sbt
ENABLE_COMET=true sbt -Dspark.test.includeSlowTests=true
@@ -74,6 +74,14 @@ ENABLE_COMET=true sbt -Dspark.test.includeSlowTests=true
```sbt
sql/testOnly org.apache.spark.sql.DynamicPartitionPruningV1SuiteAEOn -- -z
"SPARK-35568"
```
+#### Steps to run individual test suites in IntelliJ IDE
+1. Add below configuration in VM Options for your test case (apache-spark
repository)
+```sbt
+-Dspark.comet.enabled=true -Dspark.comet.debug.enabled=true
-Dspark.plugins=org.apache.spark.CometPlugin -DXmx4096m
-Dspark.executor.heartbeatInterval=20000 -Dspark.network.timeout=10000
--add-exports=java.base/sun.nio.ch=ALL-UNNAMED
--add-opens=java.base/java.nio=ALL-UNNAMED
+```
+2. Set `ENABLE_COMET=true` in environment variables
+
+3. After the above tests are configured, spark tests can be run with debugging
enabled on spark/comet code. Note that Comet is added as a dependency and the
classes are readonly while debugging from Spark. Any new changes to Comet are
to be built and deployed locally through the command (`PROFILES="-Pspark-3.4"
make release`)
## Creating a diff file for a new Spark version
diff --git a/contributor-guide/spark-sql-tests.html
b/contributor-guide/spark-sql-tests.html
index 3c9f8bcb5..6005cdfc0 100644
--- a/contributor-guide/spark-sql-tests.html
+++ b/contributor-guide/spark-sql-tests.html
@@ -311,8 +311,13 @@ under the License.
</a>
</li>
<li class="toc-h3 nav-item toc-entry">
- <a class="reference internal nav-link"
href="#steps-to-run-individual-test-suites">
- Steps to run individual test suites
+ <a class="reference internal nav-link"
href="#steps-to-run-individual-test-suites-through-sbt">
+ Steps to run individual test suites through SBT
+ </a>
+ </li>
+ <li class="toc-h3 nav-item toc-entry">
+ <a class="reference internal nav-link"
href="#steps-to-run-individual-test-suites-in-intellij-ide">
+ Steps to run individual test suites in IntelliJ IDE
</a>
</li>
</ul>
@@ -422,8 +427,8 @@ git<span class="w"> </span>apply<span class="w">
</span>../datafusion-comet/dev/
</pre></div>
</div>
</section>
-<section id="steps-to-run-individual-test-suites">
-<h3>Steps to run individual test suites<a class="headerlink"
href="#steps-to-run-individual-test-suites" title="Link to this
heading">¶</a></h3>
+<section id="steps-to-run-individual-test-suites-through-sbt">
+<h3>Steps to run individual test suites through SBT<a class="headerlink"
href="#steps-to-run-individual-test-suites-through-sbt" title="Link to this
heading">¶</a></h3>
<ol class="arabic simple">
<li><p>Open SBT with Comet enabled</p></li>
</ol>
@@ -437,6 +442,20 @@ git<span class="w"> </span>apply<span class="w">
</span>../datafusion-comet/dev/
</pre></div>
</div>
</section>
+<section id="steps-to-run-individual-test-suites-in-intellij-ide">
+<h3>Steps to run individual test suites in IntelliJ IDE<a class="headerlink"
href="#steps-to-run-individual-test-suites-in-intellij-ide" title="Link to this
heading">¶</a></h3>
+<ol class="arabic simple">
+<li><p>Add below configuration in VM Options for your test case (apache-spark
repository)</p></li>
+</ol>
+<div class="highlight-sbt notranslate"><div
class="highlight"><pre><span></span>-Dspark.comet.enabled=true
-Dspark.comet.debug.enabled=true -Dspark.plugins=org.apache.spark.CometPlugin
-DXmx4096m -Dspark.executor.heartbeatInterval=20000
-Dspark.network.timeout=10000 --add-exports=java.base/sun.nio.ch=ALL-UNNAMED
--add-opens=java.base/java.nio=ALL-UNNAMED
+</pre></div>
+</div>
+<ol class="arabic simple" start="2">
+<li><p>Set <code class="docutils literal notranslate"><span
class="pre">ENABLE_COMET=true</span></code> in environment variables
+<img alt="img.png" src="../_images/img.png" /></p></li>
+<li><p>After the above tests are configured, spark tests can be run with
debugging enabled on spark/comet code. Note that Comet is added as a dependency
and the classes are readonly while debugging from Spark. Any new changes to
Comet are to be built and deployed locally through the command (<code
class="docutils literal notranslate"><span
class="pre">PROFILES="-Pspark-3.4"</span> <span
class="pre">make</span> <span class="pre">release</span></code>)</p></li>
+</ol>
+</section>
</section>
<section id="creating-a-diff-file-for-a-new-spark-version">
<h2>Creating a diff file for a new Spark version<a class="headerlink"
href="#creating-a-diff-file-for-a-new-spark-version" title="Link to this
heading">¶</a></h2>
diff --git a/searchindex.js b/searchindex.js
index 05253655c..ea98c5371 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"1. Install Comet": [[11, "install-comet"]],
"2. Clone Spark and Apply Diff": [[11, "clone-spark-and-apply-diff"]], "3. Run
Spark SQL Tests": [[11, "run-spark-sql-tests"]], "ANSI mode": [[14,
"ansi-mode"]], "API Differences Between Spark Versions": [[0,
"api-differences-between-spark-versions"]], "ASF Links": [[13, null]],
"Accelerating Apache Iceberg Parquet Scans using Comet (Experimental)": [[19,
null]], "Adding Spark-side Tests for the New Expression": [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"1. Install Comet": [[11, "install-comet"]],
"2. Clone Spark and Apply Diff": [[11, "clone-spark-and-apply-diff"]], "3. Run
Spark SQL Tests": [[11, "run-spark-sql-tests"]], "ANSI mode": [[14,
"ansi-mode"]], "API Differences Between Spark Versions": [[0,
"api-differences-between-spark-versions"]], "ASF Links": [[13, null]],
"Accelerating Apache Iceberg Parquet Scans using Comet (Experimental)": [[19,
null]], "Adding Spark-side Tests for the New Expression": [...]
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
