This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new d7603501f Publish built docs triggered by
646dc9b1d2dabbe821445db51d2c6757f101f23a
d7603501f is described below
commit d7603501f8c248312af244163a43503b92ba92b2
Author: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
AuthorDate: Mon Mar 24 21:27:06 2025 +0000
Publish built docs triggered by 646dc9b1d2dabbe821445db51d2c6757f101f23a
---
_sources/contributor-guide/spark-sql-tests.md.txt | 6 +++---
_sources/user-guide/configs.md.txt | 6 +++---
_sources/user-guide/kubernetes.md.txt | 10 +++++-----
contributor-guide/spark-sql-tests.html | 6 +++---
searchindex.js | 2 +-
user-guide/configs.html | 6 +++---
user-guide/kubernetes.html | 10 +++++-----
7 files changed, 23 insertions(+), 23 deletions(-)
diff --git a/_sources/contributor-guide/spark-sql-tests.md.txt
b/_sources/contributor-guide/spark-sql-tests.md.txt
index de0584b6e..1776a4a86 100644
--- a/_sources/contributor-guide/spark-sql-tests.md.txt
+++ b/_sources/contributor-guide/spark-sql-tests.md.txt
@@ -72,11 +72,11 @@ of Apache Spark to enable Comet when running tests. This is
a highly manual proc
vary depending on the changes in the new version of Spark, but here is a
general guide to the process.
We typically start by applying a patch from a previous version of Spark. For
example, when enabling the tests
-for Spark version 3.5.4 we may start by applying the existing diff for 3.4.3
first.
+for Spark version 3.5.5 we may start by applying the existing diff for 3.4.3
first.
```shell
cd git/apache/spark
-git checkout v3.5.4
+git checkout v3.5.5
git apply --reject --whitespace=fix ../datafusion-comet/dev/diffs/3.4.3.diff
```
@@ -118,7 +118,7 @@ wiggle --replace
./sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.sc
## Generating The Diff File
```shell
-git diff v3.5.4 > ../datafusion-comet/dev/diffs/3.5.4.diff
+git diff v3.5.5 > ../datafusion-comet/dev/diffs/3.5.5.diff
```
## Running Tests in CI
diff --git a/_sources/user-guide/configs.md.txt
b/_sources/user-guide/configs.md.txt
index d0f95480d..529103d8b 100644
--- a/_sources/user-guide/configs.md.txt
+++ b/_sources/user-guide/configs.md.txt
@@ -71,9 +71,9 @@ Comet provides the following configuration settings.
| spark.comet.explain.verbose.enabled | When this setting is enabled, Comet
will provide a verbose tree representation of the extended information. | false
|
| spark.comet.explainFallback.enabled | When this setting is enabled, Comet
will provide logging explaining the reason(s) why a query stage cannot be
executed natively. Set this to false to reduce the amount of logging. | false |
| spark.comet.expression.allowIncompatible | Comet is not currently fully
compatible with Spark for all expressions. Set this config to true to allow
them anyway. For more information, refer to the Comet Compatibility Guide
(https://datafusion.apache.org/comet/user-guide/compatibility.html). | false |
-| spark.comet.memory.overhead.factor | Fraction of executor memory to be
allocated as additional memory for Comet when running in on-heap mode. For more
information, refer to the Comet Tuning Guide
(https://datafusion.apache.org/comet/user-guide/tuning.html). | 0.2 |
-| spark.comet.memory.overhead.min | Minimum amount of additional memory to be
allocated per executor process for Comet, in MiB, when running in on-heap mode.
For more information, refer to the Comet Tuning Guide
(https://datafusion.apache.org/comet/user-guide/tuning.html). | 402653184b |
-| spark.comet.memoryOverhead | The amount of additional memory to be allocated
per executor process for Comet, in MiB, when running in on-heap mode. This
config is optional. If this is not specified, it will be set to
`spark.comet.memory.overhead.factor` * `spark.executor.memory`. For more
information, refer to the Comet Tuning Guide
(https://datafusion.apache.org/comet/user-guide/tuning.html). | |
+| spark.comet.memory.overhead.factor | Fraction of executor memory to be
allocated as additional memory for Comet when running Spark in on-heap mode.
For more information, refer to the Comet Tuning Guide
(https://datafusion.apache.org/comet/user-guide/tuning.html). | 0.2 |
+| spark.comet.memory.overhead.min | Minimum amount of additional memory to be
allocated per executor process for Comet, in MiB, when running Spark in on-heap
mode. For more information, refer to the Comet Tuning Guide
(https://datafusion.apache.org/comet/user-guide/tuning.html). | 402653184b |
+| spark.comet.memoryOverhead | The amount of additional memory to be allocated
per executor process for Comet, in MiB, when running Spark in on-heap mode.
This config is optional. If this is not specified, it will be set to
`spark.comet.memory.overhead.factor` * `spark.executor.memory`. For more
information, refer to the Comet Tuning Guide
(https://datafusion.apache.org/comet/user-guide/tuning.html). | |
| spark.comet.metrics.updateInterval | The interval in milliseconds to update
metrics. If interval is negative, metrics will be updated upon task completion.
| 3000 |
| spark.comet.nativeLoadRequired | Whether to require Comet native library to
load successfully when Comet is enabled. If not, Comet will silently fallback
to Spark when it fails to load the native lib. Otherwise, an error will be
thrown and the Spark job will be aborted. | false |
| spark.comet.parquet.enable.directBuffer | Whether to use Java direct byte
buffer when reading Parquet. | false |
diff --git a/_sources/user-guide/kubernetes.md.txt
b/_sources/user-guide/kubernetes.md.txt
index 0d1418f83..9cb7bab86 100644
--- a/_sources/user-guide/kubernetes.md.txt
+++ b/_sources/user-guide/kubernetes.md.txt
@@ -66,10 +66,10 @@ metadata:
spec:
type: Scala
mode: cluster
- image: apache/datafusion-comet:0.7.0-spark3.5.4-scala2.12-java11
+ image: apache/datafusion-comet:0.7.0-spark3.5.5-scala2.12-java11
imagePullPolicy: IfNotPresent
mainClass: org.apache.spark.examples.SparkPi
- mainApplicationFile:
local:///opt/spark/examples/jars/spark-examples_2.12-3.5.4.jar
+ mainApplicationFile:
local:///opt/spark/examples/jars/spark-examples_2.12-3.5.5.jar
sparkConf:
"spark.executor.extraClassPath":
"/opt/spark/jars/comet-spark-spark3.5_2.12-0.7.0.jar"
"spark.driver.extraClassPath":
"/opt/spark/jars/comet-spark-spark3.5_2.12-0.7.0.jar"
@@ -80,17 +80,17 @@ spec:
"spark.comet.exec.shuffle.enabled": "true"
"spark.comet.exec.shuffle.mode": "auto"
"spark.shuffle.manager":
"org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager"
- sparkVersion: 3.5.4
+ sparkVersion: 3.5.5
driver:
labels:
- version: 3.5.4
+ version: 3.5.5
cores: 1
coreLimit: 1200m
memory: 512m
serviceAccount: spark-operator-spark
executor:
labels:
- version: 3.5.4
+ version: 3.5.5
instances: 1
cores: 1
coreLimit: 1200m
diff --git a/contributor-guide/spark-sql-tests.html
b/contributor-guide/spark-sql-tests.html
index 8441a1f4e..8500b2b36 100644
--- a/contributor-guide/spark-sql-tests.html
+++ b/contributor-guide/spark-sql-tests.html
@@ -410,9 +410,9 @@ git<span class="w"> </span>apply<span class="w">
</span>../datafusion-comet/dev/
of Apache Spark to enable Comet when running tests. This is a highly manual
process and the process can
vary depending on the changes in the new version of Spark, but here is a
general guide to the process.</p>
<p>We typically start by applying a patch from a previous version of Spark.
For example, when enabling the tests
-for Spark version 3.5.4 we may start by applying the existing diff for 3.4.3
first.</p>
+for Spark version 3.5.5 we may start by applying the existing diff for 3.4.3
first.</p>
<div class="highlight-shell notranslate"><div
class="highlight"><pre><span></span><span class="nb">cd</span><span class="w">
</span>git/apache/spark
-git<span class="w"> </span>checkout<span class="w"> </span>v3.5.4
+git<span class="w"> </span>checkout<span class="w"> </span>v3.5.5
git<span class="w"> </span>apply<span class="w"> </span>--reject<span
class="w"> </span>--whitespace<span class="o">=</span>fix<span class="w">
</span>../datafusion-comet/dev/diffs/3.4.3.diff
</pre></div>
</div>
@@ -448,7 +448,7 @@ command generated the following files.</p>
</section>
<section id="generating-the-diff-file">
<h2>Generating The Diff File<a class="headerlink"
href="#generating-the-diff-file" title="Link to this heading">ΒΆ</a></h2>
-<div class="highlight-shell notranslate"><div
class="highlight"><pre><span></span>git<span class="w"> </span>diff<span
class="w"> </span>v3.5.4<span class="w"> </span>><span class="w">
</span>../datafusion-comet/dev/diffs/3.5.4.diff
+<div class="highlight-shell notranslate"><div
class="highlight"><pre><span></span>git<span class="w"> </span>diff<span
class="w"> </span>v3.5.5<span class="w"> </span>><span class="w">
</span>../datafusion-comet/dev/diffs/3.5.5.diff
</pre></div>
</div>
</section>
diff --git a/searchindex.js b/searchindex.js
index 2ca4949ce..b94ce9d9e 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"1. Install Comet": [[9, "install-comet"]], "2.
Clone Spark and Apply Diff": [[9, "clone-spark-and-apply-diff"]], "3. Run Spark
SQL Tests": [[9, "run-spark-sql-tests"]], "ANSI mode": [[11, "ansi-mode"]],
"API Differences Between Spark Versions": [[0,
"api-differences-between-spark-versions"]], "ASF Links": [[10, null]], "Adding
Spark-side Tests for the New Expression": [[0,
"adding-spark-side-tests-for-the-new-expression"]], "Adding a New Expression":
[[0, [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"1. Install Comet": [[9, "install-comet"]], "2.
Clone Spark and Apply Diff": [[9, "clone-spark-and-apply-diff"]], "3. Run Spark
SQL Tests": [[9, "run-spark-sql-tests"]], "ANSI mode": [[11, "ansi-mode"]],
"API Differences Between Spark Versions": [[0,
"api-differences-between-spark-versions"]], "ASF Links": [[10, null]], "Adding
Spark-side Tests for the New Expression": [[0,
"adding-spark-side-tests-for-the-new-expression"]], "Adding a New Expression":
[[0, [...]
\ No newline at end of file
diff --git a/user-guide/configs.html b/user-guide/configs.html
index edd91ab80..89239836b 100644
--- a/user-guide/configs.html
+++ b/user-guide/configs.html
@@ -512,15 +512,15 @@ TO MODIFY THIS CONTENT MAKE SURE THAT YOU MAKE YOUR
CHANGES TO THE TEMPLATE FILE
<td><p>false</p></td>
</tr>
<tr class="row-even"><td><p>spark.comet.memory.overhead.factor</p></td>
-<td><p>Fraction of executor memory to be allocated as additional memory for
Comet when running in on-heap mode. For more information, refer to the Comet
Tuning Guide
(https://datafusion.apache.org/comet/user-guide/tuning.html).</p></td>
+<td><p>Fraction of executor memory to be allocated as additional memory for
Comet when running Spark in on-heap mode. For more information, refer to the
Comet Tuning Guide
(https://datafusion.apache.org/comet/user-guide/tuning.html).</p></td>
<td><p>0.2</p></td>
</tr>
<tr class="row-odd"><td><p>spark.comet.memory.overhead.min</p></td>
-<td><p>Minimum amount of additional memory to be allocated per executor
process for Comet, in MiB, when running in on-heap mode. For more information,
refer to the Comet Tuning Guide
(https://datafusion.apache.org/comet/user-guide/tuning.html).</p></td>
+<td><p>Minimum amount of additional memory to be allocated per executor
process for Comet, in MiB, when running Spark in on-heap mode. For more
information, refer to the Comet Tuning Guide
(https://datafusion.apache.org/comet/user-guide/tuning.html).</p></td>
<td><p>402653184b</p></td>
</tr>
<tr class="row-even"><td><p>spark.comet.memoryOverhead</p></td>
-<td><p>The amount of additional memory to be allocated per executor process
for Comet, in MiB, when running in on-heap mode. This config is optional. If
this is not specified, it will be set to <code class="docutils literal
notranslate"><span class="pre">spark.comet.memory.overhead.factor</span></code>
* <code class="docutils literal notranslate"><span
class="pre">spark.executor.memory</span></code>. For more information, refer to
the Comet Tuning Guide (https://datafusion.apache.org/com [...]
+<td><p>The amount of additional memory to be allocated per executor process
for Comet, in MiB, when running Spark in on-heap mode. This config is optional.
If this is not specified, it will be set to <code class="docutils literal
notranslate"><span class="pre">spark.comet.memory.overhead.factor</span></code>
* <code class="docutils literal notranslate"><span
class="pre">spark.executor.memory</span></code>. For more information, refer to
the Comet Tuning Guide (https://datafusion.apache.o [...]
<td><p></p></td>
</tr>
<tr class="row-odd"><td><p>spark.comet.metrics.updateInterval</p></td>
diff --git a/user-guide/kubernetes.html b/user-guide/kubernetes.html
index 0c10814c7..71c83e234 100644
--- a/user-guide/kubernetes.html
+++ b/user-guide/kubernetes.html
@@ -392,10 +392,10 @@ metadata:
spec:
<span class="w"> </span>type:<span class="w"> </span>Scala
<span class="w"> </span>mode:<span class="w"> </span>cluster
-<span class="w"> </span>image:<span class="w">
</span>apache/datafusion-comet:0.7.0-spark3.5.4-scala2.12-java11
+<span class="w"> </span>image:<span class="w">
</span>apache/datafusion-comet:0.7.0-spark3.5.5-scala2.12-java11
<span class="w"> </span>imagePullPolicy:<span class="w"> </span>IfNotPresent
<span class="w"> </span>mainClass:<span class="w">
</span>org.apache.spark.examples.SparkPi
-<span class="w"> </span>mainApplicationFile:<span class="w">
</span>local:///opt/spark/examples/jars/spark-examples_2.12-3.5.4.jar
+<span class="w"> </span>mainApplicationFile:<span class="w">
</span>local:///opt/spark/examples/jars/spark-examples_2.12-3.5.5.jar
<span class="w"> </span>sparkConf:
<span class="w"> </span><span
class="s2">"spark.executor.extraClassPath"</span>:<span class="w">
</span><span
class="s2">"/opt/spark/jars/comet-spark-spark3.5_2.12-0.7.0.jar"</span>
<span class="w"> </span><span
class="s2">"spark.driver.extraClassPath"</span>:<span class="w">
</span><span
class="s2">"/opt/spark/jars/comet-spark-spark3.5_2.12-0.7.0.jar"</span>
@@ -406,17 +406,17 @@ spec:
<span class="w"> </span><span
class="s2">"spark.comet.exec.shuffle.enabled"</span>:<span class="w">
</span><span class="s2">"true"</span>
<span class="w"> </span><span
class="s2">"spark.comet.exec.shuffle.mode"</span>:<span class="w">
</span><span class="s2">"auto"</span>
<span class="w"> </span><span
class="s2">"spark.shuffle.manager"</span>:<span class="w">
</span><span
class="s2">"org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager"</span>
-<span class="w"> </span>sparkVersion:<span class="w"> </span><span
class="m">3</span>.5.4
+<span class="w"> </span>sparkVersion:<span class="w"> </span><span
class="m">3</span>.5.5
<span class="w"> </span>driver:
<span class="w"> </span>labels:
-<span class="w"> </span>version:<span class="w"> </span><span
class="m">3</span>.5.4
+<span class="w"> </span>version:<span class="w"> </span><span
class="m">3</span>.5.5
<span class="w"> </span>cores:<span class="w"> </span><span
class="m">1</span>
<span class="w"> </span>coreLimit:<span class="w"> </span>1200m
<span class="w"> </span>memory:<span class="w"> </span>512m
<span class="w"> </span>serviceAccount:<span class="w">
</span>spark-operator-spark
<span class="w"> </span>executor:
<span class="w"> </span>labels:
-<span class="w"> </span>version:<span class="w"> </span><span
class="m">3</span>.5.4
+<span class="w"> </span>version:<span class="w"> </span><span
class="m">3</span>.5.5
<span class="w"> </span>instances:<span class="w"> </span><span
class="m">1</span>
<span class="w"> </span>cores:<span class="w"> </span><span
class="m">1</span>
<span class="w"> </span>coreLimit:<span class="w"> </span>1200m
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
