This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 21aae75b Publish built docs triggered by
2c832b4a56eafa3dacbe3ef31d99adabccb803bf
21aae75b is described below
commit 21aae75b1a4e213e56605b3fe02384090222e7b9
Author: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
AuthorDate: Thu Nov 14 19:56:33 2024 +0000
Publish built docs triggered by 2c832b4a56eafa3dacbe3ef31d99adabccb803bf
---
_sources/user-guide/tuning.md.txt | 32 ++------------------------------
searchindex.js | 2 +-
user-guide/tuning.html | 36 ++----------------------------------
3 files changed, 5 insertions(+), 65 deletions(-)
diff --git a/_sources/user-guide/tuning.md.txt
b/_sources/user-guide/tuning.md.txt
index 30ada4c9..b1838ca8 100644
--- a/_sources/user-guide/tuning.md.txt
+++ b/_sources/user-guide/tuning.md.txt
@@ -23,40 +23,12 @@ Comet provides some tuning options to help you get the best
performance from you
## Memory Tuning
-Comet provides two options for memory management:
-
-- **Unified Memory Management** shares an off-heap memory pool between Spark
and Comet. This is the recommended option.
-- **Native Memory Management** leverages DataFusion's memory management for
the native plans and allocates memory independently of Spark.
-
-### Unified Memory Management
-
-This option is automatically enabled when `spark.memory.offHeap.enabled=true`.
+Comet shares an off-heap memory pool between Spark and Comet. This requires
setting `spark.memory.offHeap.enabled=true`.
+If this setting is not enabled, Comet will not accelerate queries and will
fall back to Spark.
Each executor will have a single memory pool which will be shared by all
native plans being executed within that
process, and by Spark itself. The size of the pool is specified by
`spark.memory.offHeap.size`.
-### Native Memory Management
-
-This option is automatically enabled when `spark.memory.offHeap.enabled=false`.
-
-Each native plan has a dedicated memory pool.
-
-By default, the size of each pool is `spark.comet.memory.overhead.factor *
spark.executor.memory`. The default value
-for `spark.comet.memory.overhead.factor` is `0.2`.
-
-It is important to take executor concurrency into account. The maximum number
of concurrent plans in an executor can
-be calculated with `spark.executor.cores / spark.task.cpus`.
-
-For example, if the executor can execute 4 plans concurrently, then the total
amount of memory allocated will be
-`4 * spark.comet.memory.overhead.factor * spark.executor.memory`.
-
-It is also possible to set `spark.comet.memoryOverhead` to the desired size
for each pool, rather than calculating
-it based on `spark.comet.memory.overhead.factor`.
-
-If both `spark.comet.memoryOverhead` and `spark.comet.memory.overhead.factor`
are set, the former will be used.
-
-Comet will allocate at least `spark.comet.memory.overhead.min` memory per pool.
-
### Determining How Much Memory to Allocate
Generally, increasing memory overhead will improve query performance,
especially for queries containing joins and
diff --git a/searchindex.js b/searchindex.js
index e70c8491..e605b496 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"1. Install Comet": [[9, "install-comet"]], "2.
Clone Spark and Apply Diff": [[9, "clone-spark-and-apply-diff"]], "3. Run Spark
SQL Tests": [[9, "run-spark-sql-tests"]], "ANSI mode": [[11, "ansi-mode"]],
"API Differences Between Spark Versions": [[0,
"api-differences-between-spark-versions"]], "ASF Links": [[10, null]], "Adding
Spark-side Tests for the New Expression": [[0,
"adding-spark-side-tests-for-the-new-expression"]], "Adding a New Expression":
[[0, [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"1. Install Comet": [[9, "install-comet"]], "2.
Clone Spark and Apply Diff": [[9, "clone-spark-and-apply-diff"]], "3. Run Spark
SQL Tests": [[9, "run-spark-sql-tests"]], "ANSI mode": [[11, "ansi-mode"]],
"API Differences Between Spark Versions": [[0,
"api-differences-between-spark-versions"]], "ASF Links": [[10, null]], "Adding
Spark-side Tests for the New Expression": [[0,
"adding-spark-side-tests-for-the-new-expression"]], "Adding a New Expression":
[[0, [...]
\ No newline at end of file
diff --git a/user-guide/tuning.html b/user-guide/tuning.html
index 2f226a38..0b1074cc 100644
--- a/user-guide/tuning.html
+++ b/user-guide/tuning.html
@@ -286,16 +286,6 @@ under the License.
Memory Tuning
</a>
<ul class="nav section-nav flex-column">
- <li class="toc-h3 nav-item toc-entry">
- <a class="reference internal nav-link" href="#unified-memory-management">
- Unified Memory Management
- </a>
- </li>
- <li class="toc-h3 nav-item toc-entry">
- <a class="reference internal nav-link" href="#native-memory-management">
- Native Memory Management
- </a>
- </li>
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link"
href="#determining-how-much-memory-to-allocate">
Determining How Much Memory to Allocate
@@ -398,32 +388,10 @@ under the License.
<p>Comet provides some tuning options to help you get the best performance
from your queries.</p>
<section id="memory-tuning">
<h2>Memory Tuning<a class="headerlink" href="#memory-tuning" title="Link to
this heading">¶</a></h2>
-<p>Comet provides two options for memory management:</p>
-<ul class="simple">
-<li><p><strong>Unified Memory Management</strong> shares an off-heap memory
pool between Spark and Comet. This is the recommended option.</p></li>
-<li><p><strong>Native Memory Management</strong> leverages DataFusion’s memory
management for the native plans and allocates memory independently of
Spark.</p></li>
-</ul>
-<section id="unified-memory-management">
-<h3>Unified Memory Management<a class="headerlink"
href="#unified-memory-management" title="Link to this heading">¶</a></h3>
-<p>This option is automatically enabled when <code class="docutils literal
notranslate"><span
class="pre">spark.memory.offHeap.enabled=true</span></code>.</p>
+<p>Comet shares an off-heap memory pool between Spark and Comet. This requires
setting <code class="docutils literal notranslate"><span
class="pre">spark.memory.offHeap.enabled=true</span></code>.
+If this setting is not enabled, Comet will not accelerate queries and will
fall back to Spark.</p>
<p>Each executor will have a single memory pool which will be shared by all
native plans being executed within that
process, and by Spark itself. The size of the pool is specified by <code
class="docutils literal notranslate"><span
class="pre">spark.memory.offHeap.size</span></code>.</p>
-</section>
-<section id="native-memory-management">
-<h3>Native Memory Management<a class="headerlink"
href="#native-memory-management" title="Link to this heading">¶</a></h3>
-<p>This option is automatically enabled when <code class="docutils literal
notranslate"><span
class="pre">spark.memory.offHeap.enabled=false</span></code>.</p>
-<p>Each native plan has a dedicated memory pool.</p>
-<p>By default, the size of each pool is <code class="docutils literal
notranslate"><span class="pre">spark.comet.memory.overhead.factor</span> <span
class="pre">*</span> <span class="pre">spark.executor.memory</span></code>. The
default value
-for <code class="docutils literal notranslate"><span
class="pre">spark.comet.memory.overhead.factor</span></code> is <code
class="docutils literal notranslate"><span class="pre">0.2</span></code>.</p>
-<p>It is important to take executor concurrency into account. The maximum
number of concurrent plans in an executor can
-be calculated with <code class="docutils literal notranslate"><span
class="pre">spark.executor.cores</span> <span class="pre">/</span> <span
class="pre">spark.task.cpus</span></code>.</p>
-<p>For example, if the executor can execute 4 plans concurrently, then the
total amount of memory allocated will be
-<code class="docutils literal notranslate"><span class="pre">4</span> <span
class="pre">*</span> <span
class="pre">spark.comet.memory.overhead.factor</span> <span
class="pre">*</span> <span class="pre">spark.executor.memory</span></code>.</p>
-<p>It is also possible to set <code class="docutils literal notranslate"><span
class="pre">spark.comet.memoryOverhead</span></code> to the desired size for
each pool, rather than calculating
-it based on <code class="docutils literal notranslate"><span
class="pre">spark.comet.memory.overhead.factor</span></code>.</p>
-<p>If both <code class="docutils literal notranslate"><span
class="pre">spark.comet.memoryOverhead</span></code> and <code class="docutils
literal notranslate"><span
class="pre">spark.comet.memory.overhead.factor</span></code> are set, the
former will be used.</p>
-<p>Comet will allocate at least <code class="docutils literal
notranslate"><span class="pre">spark.comet.memory.overhead.min</span></code>
memory per pool.</p>
-</section>
<section id="determining-how-much-memory-to-allocate">
<h3>Determining How Much Memory to Allocate<a class="headerlink"
href="#determining-how-much-memory-to-allocate" title="Link to this
heading">¶</a></h3>
<p>Generally, increasing memory overhead will improve query performance,
especially for queries containing joins and
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
