This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 2f85a0550 Publish built docs triggered by
1ec35634cfb30936a5034f19c9aab6290abf9b9b
2f85a0550 is described below
commit 2f85a05508b17b0f266d10fc5b468176886b4100
Author: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
AuthorDate: Wed Nov 26 13:47:52 2025 +0000
Publish built docs triggered by 1ec35634cfb30936a5034f19c9aab6290abf9b9b
---
_sources/user-guide/latest/configs.md.txt | 1 +
_sources/user-guide/latest/operators.md.txt | 41 +++++++++++++++--------------
searchindex.js | 2 +-
user-guide/latest/configs.html | 10 ++++---
user-guide/latest/operators.html | 24 ++++++++++-------
5 files changed, 44 insertions(+), 34 deletions(-)
diff --git a/_sources/user-guide/latest/configs.md.txt
b/_sources/user-guide/latest/configs.md.txt
index 1e77032f7..a1c3212c2 100644
--- a/_sources/user-guide/latest/configs.md.txt
+++ b/_sources/user-guide/latest/configs.md.txt
@@ -142,6 +142,7 @@ These settings can be used to determine which parts of the
plan are accelerated
| `spark.comet.exec.onHeap.enabled` | Whether to allow Comet to run in on-heap
mode. Required for running Spark SQL tests. It can be overridden by the
environment variable `ENABLE_COMET_ONHEAP`. | false |
| `spark.comet.exec.onHeap.memoryPool` | The type of memory pool to be used
for Comet native execution when running Spark in on-heap mode. Available pool
types are `greedy`, `fair_spill`, `greedy_task_shared`,
`fair_spill_task_shared`, `greedy_global`, `fair_spill_global`, and
`unbounded`. | greedy_task_shared |
| `spark.comet.memoryOverhead` | The amount of additional memory to be
allocated per executor process for Comet, in MiB, when running Spark in on-heap
mode. | 1024 MiB |
+| `spark.comet.parquet.write.enabled` | Whether to enable native Parquet write
through Comet. When enabled, Comet will intercept Parquet write operations and
execute them natively. This feature is highly experimental and only partially
implemented. It should not be used in production. | false |
| `spark.comet.sparkToColumnar.enabled` | Whether to enable Spark to Arrow
columnar conversion. When this is turned on, Comet will convert operators in
`spark.comet.sparkToColumnar.supportedOperatorList` into Arrow columnar format
before processing. This is an experimental feature and has known issues with
non-UTC timezones. | false |
| `spark.comet.sparkToColumnar.supportedOperatorList` | A comma-separated list
of operators that will be converted to Arrow columnar format when
`spark.comet.sparkToColumnar.enabled` is true. |
Range,InMemoryTableScan,RDDScan |
| `spark.comet.testing.strict` | Experimental option to enable strict testing,
which will fail tests that could be more comprehensive, such as checking for a
specific fallback reason. It can be overridden by the environment variable
`ENABLE_COMET_STRICT_TESTING`. | false |
diff --git a/_sources/user-guide/latest/operators.md.txt
b/_sources/user-guide/latest/operators.md.txt
index fdfbcef68..f5f2d9724 100644
--- a/_sources/user-guide/latest/operators.md.txt
+++ b/_sources/user-guide/latest/operators.md.txt
@@ -22,25 +22,26 @@
The following Spark operators are currently replaced with native versions.
Query stages that contain any operators
not supported by Comet will fall back to regular Spark execution.
-| Operator | Spark-Compatible? | Compatibility Notes
|
-| ----------------------- | ----------------- |
------------------------------------------------------------------------------------------------------------------
|
-| BatchScanExec | Yes | Supports Parquet files and
Apache Iceberg Parquet scans. See the [Comet Compatibility Guide] for more
information. |
-| BroadcastExchangeExec | Yes |
|
-| BroadcastHashJoinExec | Yes |
|
-| ExpandExec | Yes |
|
-| FileSourceScanExec | Yes | Supports Parquet files. See
the [Comet Compatibility Guide] for more information.
|
-| FilterExec | Yes |
|
-| GlobalLimitExec | Yes |
|
-| HashAggregateExec | Yes |
|
-| LocalLimitExec | Yes |
|
-| LocalTableScanExec | No | Experimental and disabled by
default.
|
-| ObjectHashAggregateExec | Yes | Supports a limited number of
aggregates, such as `bloom_filter_agg`.
|
-| ProjectExec | Yes |
|
-| ShuffleExchangeExec | Yes |
|
-| ShuffledHashJoinExec | Yes |
|
-| SortExec | Yes |
|
-| SortMergeJoinExec | Yes |
|
-| UnionExec | Yes |
|
-| WindowExec | No | Disabled by default due to
known correctness issues.
|
+| Operator | Spark-Compatible? | Compatibility Notes
|
+| --------------------------------- | ----------------- |
------------------------------------------------------------------------------------------------------------------
|
+| BatchScanExec | Yes | Supports Parquet
files and Apache Iceberg Parquet scans. See the [Comet Compatibility Guide] for
more information. |
+| BroadcastExchangeExec | Yes |
|
+| BroadcastHashJoinExec | Yes |
|
+| ExpandExec | Yes |
|
+| FileSourceScanExec | Yes | Supports Parquet
files. See the [Comet Compatibility Guide] for more information.
|
+| FilterExec | Yes |
|
+| GlobalLimitExec | Yes |
|
+| HashAggregateExec | Yes |
|
+| InsertIntoHadoopFsRelationCommand | No | Experimental support
for native Parquet writes. Disabled by default.
|
+| LocalLimitExec | Yes |
|
+| LocalTableScanExec | No | Experimental and
disabled by default.
|
+| ObjectHashAggregateExec | Yes | Supports a limited
number of aggregates, such as `bloom_filter_agg`.
|
+| ProjectExec | Yes |
|
+| ShuffleExchangeExec | Yes |
|
+| ShuffledHashJoinExec | Yes |
|
+| SortExec | Yes |
|
+| SortMergeJoinExec | Yes |
|
+| UnionExec | Yes |
|
+| WindowExec | No | Disabled by default
due to known correctness issues.
|
[Comet Compatibility Guide]: compatibility.md
diff --git a/searchindex.js b/searchindex.js
index 7b08b6367..5e6fe9e11 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"1. Install Comet": [[19, "install-comet"]],
"1. Native Operators (nativeExecs map)": [[4,
"native-operators-nativeexecs-map"]], "2. Clone Spark and Apply Diff": [[19,
"clone-spark-and-apply-diff"]], "2. Sink Operators (sinks map)": [[4,
"sink-operators-sinks-map"]], "3. Comet JVM Operators": [[4,
"comet-jvm-operators"]], "3. Run Spark SQL Tests": [[19,
"run-spark-sql-tests"]], "ANSI Mode": [[22, "ansi-mode"], [35, "ansi-mode"],
[48, "ansi-mode"], [88, "ans [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"1. Install Comet": [[19, "install-comet"]],
"1. Native Operators (nativeExecs map)": [[4,
"native-operators-nativeexecs-map"]], "2. Clone Spark and Apply Diff": [[19,
"clone-spark-and-apply-diff"]], "2. Sink Operators (sinks map)": [[4,
"sink-operators-sinks-map"]], "3. Comet JVM Operators": [[4,
"comet-jvm-operators"]], "3. Run Spark SQL Tests": [[19,
"run-spark-sql-tests"]], "ANSI Mode": [[22, "ansi-mode"], [35, "ansi-mode"],
[48, "ansi-mode"], [88, "ans [...]
\ No newline at end of file
diff --git a/user-guide/latest/configs.html b/user-guide/latest/configs.html
index f087f7315..0dbc6e46b 100644
--- a/user-guide/latest/configs.html
+++ b/user-guide/latest/configs.html
@@ -798,15 +798,19 @@ under the License.
<td><p>The amount of additional memory to be allocated per executor process
for Comet, in MiB, when running Spark in on-heap mode.</p></td>
<td><p>1024 MiB</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.sparkToColumnar.enabled</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.parquet.write.enabled</span></code></p></td>
+<td><p>Whether to enable native Parquet write through Comet. When enabled,
Comet will intercept Parquet write operations and execute them natively. This
feature is highly experimental and only partially implemented. It should not be
used in production.</p></td>
+<td><p>false</p></td>
+</tr>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.sparkToColumnar.enabled</span></code></p></td>
<td><p>Whether to enable Spark to Arrow columnar conversion. When this is
turned on, Comet will convert operators in <code class="docutils literal
notranslate"><span
class="pre">spark.comet.sparkToColumnar.supportedOperatorList</span></code>
into Arrow columnar format before processing. This is an experimental feature
and has known issues with non-UTC timezones.</p></td>
<td><p>false</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.sparkToColumnar.supportedOperatorList</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.sparkToColumnar.supportedOperatorList</span></code></p></td>
<td><p>A comma-separated list of operators that will be converted to Arrow
columnar format when <code class="docutils literal notranslate"><span
class="pre">spark.comet.sparkToColumnar.enabled</span></code> is true.</p></td>
<td><p>Range,InMemoryTableScan,RDDScan</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.testing.strict</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.testing.strict</span></code></p></td>
<td><p>Experimental option to enable strict testing, which will fail tests
that could be more comprehensive, such as checking for a specific fallback
reason. It can be overridden by the environment variable <code class="docutils
literal notranslate"><span
class="pre">ENABLE_COMET_STRICT_TESTING</span></code>.</p></td>
<td><p>false</p></td>
</tr>
diff --git a/user-guide/latest/operators.html b/user-guide/latest/operators.html
index 861630f2e..689cea81d 100644
--- a/user-guide/latest/operators.html
+++ b/user-guide/latest/operators.html
@@ -503,43 +503,47 @@ not supported by Comet will fall back to regular Spark
execution.</p>
<td><p>Yes</p></td>
<td><p></p></td>
</tr>
-<tr class="row-even"><td><p>LocalLimitExec</p></td>
+<tr class="row-even"><td><p>InsertIntoHadoopFsRelationCommand</p></td>
+<td><p>No</p></td>
+<td><p>Experimental support for native Parquet writes. Disabled by
default.</p></td>
+</tr>
+<tr class="row-odd"><td><p>LocalLimitExec</p></td>
<td><p>Yes</p></td>
<td><p></p></td>
</tr>
-<tr class="row-odd"><td><p>LocalTableScanExec</p></td>
+<tr class="row-even"><td><p>LocalTableScanExec</p></td>
<td><p>No</p></td>
<td><p>Experimental and disabled by default.</p></td>
</tr>
-<tr class="row-even"><td><p>ObjectHashAggregateExec</p></td>
+<tr class="row-odd"><td><p>ObjectHashAggregateExec</p></td>
<td><p>Yes</p></td>
<td><p>Supports a limited number of aggregates, such as <code class="docutils
literal notranslate"><span class="pre">bloom_filter_agg</span></code>.</p></td>
</tr>
-<tr class="row-odd"><td><p>ProjectExec</p></td>
+<tr class="row-even"><td><p>ProjectExec</p></td>
<td><p>Yes</p></td>
<td><p></p></td>
</tr>
-<tr class="row-even"><td><p>ShuffleExchangeExec</p></td>
+<tr class="row-odd"><td><p>ShuffleExchangeExec</p></td>
<td><p>Yes</p></td>
<td><p></p></td>
</tr>
-<tr class="row-odd"><td><p>ShuffledHashJoinExec</p></td>
+<tr class="row-even"><td><p>ShuffledHashJoinExec</p></td>
<td><p>Yes</p></td>
<td><p></p></td>
</tr>
-<tr class="row-even"><td><p>SortExec</p></td>
+<tr class="row-odd"><td><p>SortExec</p></td>
<td><p>Yes</p></td>
<td><p></p></td>
</tr>
-<tr class="row-odd"><td><p>SortMergeJoinExec</p></td>
+<tr class="row-even"><td><p>SortMergeJoinExec</p></td>
<td><p>Yes</p></td>
<td><p></p></td>
</tr>
-<tr class="row-even"><td><p>UnionExec</p></td>
+<tr class="row-odd"><td><p>UnionExec</p></td>
<td><p>Yes</p></td>
<td><p></p></td>
</tr>
-<tr class="row-odd"><td><p>WindowExec</p></td>
+<tr class="row-even"><td><p>WindowExec</p></td>
<td><p>No</p></td>
<td><p>Disabled by default due to known correctness issues.</p></td>
</tr>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
