This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new d474698 Publish built docs triggered by
3599acb99622711affc46f15b1dcb5d35d6e9e97
d474698 is described below
commit d474698e4600efa80a2975cbaaa602fe7d9e0b96
Author: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
AuthorDate: Mon Apr 29 17:29:34 2024 +0000
Publish built docs triggered by 3599acb99622711affc46f15b1dcb5d35d6e9e97
---
_images/comet-overview.png | Bin 0 -> 172624 bytes
_images/comet-system-diagram.png | Bin 0 -> 30027 bytes
_sources/index.rst.txt | 6 +-
_sources/user-guide/datatypes.md.txt | 41 ++
_sources/user-guide/installation.md.txt | 125 ++++++
_sources/user-guide/operators.md.txt | 33 ++
_sources/user-guide/overview.md.txt | 55 +++
_static/images/comet-overview.png | Bin 0 -> 172624 bytes
_static/images/comet-plan.png | Bin 0 -> 310869 bytes
_static/images/comet-system-diagram.png | Bin 0 -> 30027 bytes
contributor-guide/contributing.html | 20 +
contributor-guide/debugging.html | 20 +
contributor-guide/development.html | 20 +
genindex.html | 20 +
index.html | 30 +-
objects.inv | Bin 405 -> 479 bytes
search.html | 20 +
searchindex.js | 2 +-
user-guide/compatibility.html | 26 +-
user-guide/{expressions.html => datatypes.html} | 159 +++-----
user-guide/expressions.html | 32 +-
user-guide/installation.html | 469 +++++++++++++++++++++++
user-guide/{expressions.html => operators.html} | 164 ++------
user-guide/{compatibility.html => overview.html} | 111 ++++--
24 files changed, 1062 insertions(+), 291 deletions(-)
diff --git a/_images/comet-overview.png b/_images/comet-overview.png
new file mode 100644
index 0000000..a3148d8
Binary files /dev/null and b/_images/comet-overview.png differ
diff --git a/_images/comet-system-diagram.png b/_images/comet-system-diagram.png
new file mode 100644
index 0000000..e7c9075
Binary files /dev/null and b/_images/comet-system-diagram.png differ
diff --git a/_sources/index.rst.txt b/_sources/index.rst.txt
index 4462a8d..a19f642 100644
--- a/_sources/index.rst.txt
+++ b/_sources/index.rst.txt
@@ -40,8 +40,12 @@ as a native runtime to achieve improvement in terms of query
efficiency and quer
:maxdepth: 1
:caption: User Guide
+ Comet Overview <user-guide/overview>
+ Installing Comet <user-guide/installation>
Supported Expressions <user-guide/expressions>
- user-guide/compatibility
+ Supported Operators <user-guide/operators>
+ Supported Data Types <user-guide/datatypes>
+ Compatibility Guide <user-guide/compatibility>
.. _toc.links:
.. toctree::
diff --git a/_sources/user-guide/datatypes.md.txt
b/_sources/user-guide/datatypes.md.txt
new file mode 100644
index 0000000..02e968a
--- /dev/null
+++ b/_sources/user-guide/datatypes.md.txt
@@ -0,0 +1,41 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Supported Spark Data Types
+
+The following Spark data types are currently available:
+
+<!-- based on org.apache.comet.serde.QueryPlanSerde.supportedDataType -->
+
+- Primitives
+ - Boolean
+ - Byte
+ - Short
+ - Integer
+ - Long
+ - Float
+ - Double
+- String
+- Binary
+- Decimal
+- Temporal
+ - Date
+ - Timestamp
+ - TimestampNTZ
+- Null
diff --git a/_sources/user-guide/installation.md.txt
b/_sources/user-guide/installation.md.txt
new file mode 100644
index 0000000..b948d50
--- /dev/null
+++ b/_sources/user-guide/installation.md.txt
@@ -0,0 +1,125 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Installing DataFusion Comet
+
+Make sure the following requirements are met and software installed on your
machine.
+
+## Supported Platforms
+
+- Linux
+- Apple OSX (Intel and Apple Silicon)
+
+## Requirements
+
+- Apache Spark 3.2, 3.3, or 3.4
+- JDK 8 and up
+- GLIBC 2.17 (Centos 7) and up
+
+## Using a Published Release
+
+There are no public releases available yet, so it is necessary to build from
source as described in the next section.
+
+## Building From Source
+
+Clone the repository:
+
+```commandline
+git clone https://github.com/apache/datafusion-comet.git
+```
+
+Build Comet for a specific Spark version:
+
+```commandline
+cd datafusion-comet
+make release PROFILES="-Pspark-3.4"
+```
+
+Note that the project builds for Scala 2.12 by default but can be built for
Scala 2.13 using an additional profile:
+
+```commandline
+make release PROFILES="-Pspark-3.4 -Pscala-2.13"
+```
+
+## Run Spark with Comet enabled
+
+Make sure `SPARK_HOME` points to the same Spark version as Comet was built for.
+
+```commandline
+$SPARK_HOME/bin/spark-shell \
+ --jars spark/target/comet-spark-spark3.4_2.12-0.1.0-SNAPSHOT.jar \
+ --conf spark.sql.extensions=org.apache.comet.CometSparkSessionExtensions \
+ --conf spark.comet.enabled=true \
+ --conf spark.comet.exec.enabled=true \
+ --conf spark.comet.exec.all.enabled=true
+```
+
+### Verify Comet enabled for Spark SQL query
+
+Create a test Parquet source
+
+```scala
+scala> (0 until 10).toDF("a").write.mode("overwrite").parquet("/tmp/test")
+```
+
+Query the data from the test source and check:
+
+- INFO message shows the native Comet library has been initialized.
+- The query plan reflects Comet operators being used for this query instead of
Spark ones
+
+```scala
+scala> spark.read.parquet("/tmp/test").createOrReplaceTempView("t1")
+scala> spark.sql("select * from t1 where a > 5").explain
+INFO src/lib.rs: Comet native library initialized
+== Physical Plan ==
+ *(1) ColumnarToRow
+ +- CometFilter [a#14], (isnotnull(a#14) AND (a#14 > 5))
+ +- CometScan parquet [a#14] Batched: true, DataFilters:
[isnotnull(a#14), (a#14 > 5)],
+ Format: CometParquet, Location: InMemoryFileIndex(1
paths)[file:/tmp/test], PartitionFilters: [],
+ PushedFilters: [IsNotNull(a), GreaterThan(a,5)], ReadSchema:
struct<a:int>
+```
+
+### Enable Comet shuffle
+
+Comet shuffle feature is disabled by default. To enable it, please add related
configs:
+
+```
+--conf
spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager
+--conf spark.comet.exec.shuffle.enabled=true
+```
+
+Above configs enable Comet native shuffle which only supports hash partition
and single partition.
+Comet native shuffle doesn't support complex types yet.
+
+Comet doesn't have official release yet so currently the only way to test it
is to build jar and include it in your
+Spark application. Depending on your deployment mode you may also need to set
the driver & executor class path(s) to
+explicitly contain Comet otherwise Spark may use a different class-loader for
the Comet components than its internal
+components which will then fail at runtime. For example:
+
+```
+--driver-class-path spark/target/comet-spark-spark3.4_2.12-0.1.0-SNAPSHOT.jar
+```
+
+Some cluster managers may require additional configuration, see
https://spark.apache.org/docs/latest/cluster-overview.html
+
+To enable columnar shuffle which supports all partitioning and basic complex
types, one more config is required:
+
+```
+--conf spark.comet.columnar.shuffle.enabled=true
+```
diff --git a/_sources/user-guide/operators.md.txt
b/_sources/user-guide/operators.md.txt
new file mode 100644
index 0000000..ec82e9f
--- /dev/null
+++ b/_sources/user-guide/operators.md.txt
@@ -0,0 +1,33 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Supported Spark Operators
+
+The following Spark operators are currently available:
+
+- FileSourceScanExec/BatchScanExec for Parquet
+- Projection
+- Filter
+- Sort
+- Hash Aggregate
+- Limit
+- Sort-merge Join
+- Hash Join
+- Shuffle
+- Expand
diff --git a/_sources/user-guide/overview.md.txt
b/_sources/user-guide/overview.md.txt
new file mode 100644
index 0000000..ff73176
--- /dev/null
+++ b/_sources/user-guide/overview.md.txt
@@ -0,0 +1,55 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Comet Overview
+
+Comet runs Spark SQL queries using the native Apache DataFusion runtime, which
is
+typically faster and more resource efficient than JVM based runtimes.
+
+
+
+Comet aims to support:
+
+- a native Parquet implementation, including both reader and writer
+- full implementation of Spark operators, including
+ Filter/Project/Aggregation/Join/Exchange etc.
+- full implementation of Spark built-in expressions
+- a UDF framework for users to migrate their existing UDF to native
+
+## Architecture
+
+The following diagram illustrates the architecture of Comet:
+
+
+
+## Current Status
+
+The project is currently integrated into Apache Spark 3.2, 3.3, and 3.4.
+
+## Feature Parity with Apache Spark
+
+The project strives to keep feature parity with Apache Spark, that is,
+users should expect the same behavior (w.r.t features, configurations,
+query results, etc) with Comet turned on or turned off in their Spark
+jobs. In addition, Comet extension should automatically detect unsupported
+features and fallback to Spark engine.
+
+To achieve this, besides unit tests within Comet itself, we also re-use
+Spark SQL tests and make sure they all pass with Comet extension
+enabled.
diff --git a/_static/images/comet-overview.png
b/_static/images/comet-overview.png
new file mode 100644
index 0000000..a3148d8
Binary files /dev/null and b/_static/images/comet-overview.png differ
diff --git a/_static/images/comet-plan.png b/_static/images/comet-plan.png
new file mode 100644
index 0000000..38c4783
Binary files /dev/null and b/_static/images/comet-plan.png differ
diff --git a/_static/images/comet-system-diagram.png
b/_static/images/comet-system-diagram.png
new file mode 100644
index 0000000..e7c9075
Binary files /dev/null and b/_static/images/comet-system-diagram.png differ
diff --git a/contributor-guide/contributing.html
b/contributor-guide/contributing.html
index a6f1e02..65d700a 100644
--- a/contributor-guide/contributing.html
+++ b/contributor-guide/contributing.html
@@ -108,11 +108,31 @@ under the License.
</span>
</p>
<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/overview.html">
+ Comet Overview
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/installation.html">
+ Installing Comet
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="../user-guide/expressions.html">
Supported Expressions
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/operators.html">
+ Supported Operators
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/datatypes.html">
+ Supported Data Types
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="../user-guide/compatibility.html">
Compatibility Guide
diff --git a/contributor-guide/debugging.html b/contributor-guide/debugging.html
index 48d5e77..a815309 100644
--- a/contributor-guide/debugging.html
+++ b/contributor-guide/debugging.html
@@ -107,11 +107,31 @@ under the License.
</span>
</p>
<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/overview.html">
+ Comet Overview
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/installation.html">
+ Installing Comet
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="../user-guide/expressions.html">
Supported Expressions
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/operators.html">
+ Supported Operators
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/datatypes.html">
+ Supported Data Types
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="../user-guide/compatibility.html">
Compatibility Guide
diff --git a/contributor-guide/development.html
b/contributor-guide/development.html
index 7b27461..7fd3dd8 100644
--- a/contributor-guide/development.html
+++ b/contributor-guide/development.html
@@ -108,11 +108,31 @@ under the License.
</span>
</p>
<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/overview.html">
+ Comet Overview
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/installation.html">
+ Installing Comet
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="../user-guide/expressions.html">
Supported Expressions
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/operators.html">
+ Supported Operators
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/datatypes.html">
+ Supported Data Types
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="../user-guide/compatibility.html">
Compatibility Guide
diff --git a/genindex.html b/genindex.html
index 8a16123..ba9c7bc 100644
--- a/genindex.html
+++ b/genindex.html
@@ -105,11 +105,31 @@ under the License.
</span>
</p>
<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="user-guide/overview.html">
+ Comet Overview
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="user-guide/installation.html">
+ Installing Comet
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="user-guide/expressions.html">
Supported Expressions
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="user-guide/operators.html">
+ Supported Operators
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="user-guide/datatypes.html">
+ Supported Data Types
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="user-guide/compatibility.html">
Compatibility Guide
diff --git a/index.html b/index.html
index cc8a388..4ac8053 100644
--- a/index.html
+++ b/index.html
@@ -53,7 +53,7 @@ under the License.
<script async="true" defer="true"
src="https://buttons.github.io/buttons.js";></script>
<link rel="index" title="Index" href="genindex.html" />
<link rel="search" title="Search" href="search.html" />
- <link rel="next" title="Supported Spark Expressions"
href="user-guide/expressions.html" />
+ <link rel="next" title="Comet Overview" href="user-guide/overview.html" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="docsearch:language" content="en">
@@ -107,11 +107,31 @@ under the License.
</span>
</p>
<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="user-guide/overview.html">
+ Comet Overview
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="user-guide/installation.html">
+ Installing Comet
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="user-guide/expressions.html">
Supported Expressions
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="user-guide/operators.html">
+ Supported Operators
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="user-guide/datatypes.html">
+ Supported Data Types
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="user-guide/compatibility.html">
Compatibility Guide
@@ -246,7 +266,11 @@ as a native runtime to achieve improvement in terms of
query efficiency and quer
<div class="toctree-wrapper compound" id="toc-links">
<p aria-level="2" class="caption" role="heading"><span
class="caption-text">User Guide</span></p>
<ul>
+<li class="toctree-l1"><a class="reference internal"
href="user-guide/overview.html">Comet Overview</a></li>
+<li class="toctree-l1"><a class="reference internal"
href="user-guide/installation.html">Installing Comet</a></li>
<li class="toctree-l1"><a class="reference internal"
href="user-guide/expressions.html">Supported Expressions</a></li>
+<li class="toctree-l1"><a class="reference internal"
href="user-guide/operators.html">Supported Operators</a></li>
+<li class="toctree-l1"><a class="reference internal"
href="user-guide/datatypes.html">Supported Data Types</a></li>
<li class="toctree-l1"><a class="reference internal"
href="user-guide/compatibility.html">Compatibility Guide</a></li>
</ul>
</div>
@@ -279,10 +303,10 @@ as a native runtime to achieve improvement in terms of
query efficiency and quer
<!-- Previous / next buttons -->
<div class='prev-next-area'>
- <a class='right-next' id="next-link" href="user-guide/expressions.html"
title="next page">
+ <a class='right-next' id="next-link" href="user-guide/overview.html"
title="next page">
<div class="prev-next-info">
<p class="prev-next-subtitle">next</p>
- <p class="prev-next-title">Supported Spark Expressions</p>
+ <p class="prev-next-title">Comet Overview</p>
</div>
<i class="fas fa-angle-right"></i>
</a>
diff --git a/objects.inv b/objects.inv
index fc585e9..f82e77b 100644
Binary files a/objects.inv and b/objects.inv differ
diff --git a/search.html b/search.html
index 7939299..4e8a4ca 100644
--- a/search.html
+++ b/search.html
@@ -112,11 +112,31 @@ under the License.
</span>
</p>
<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="user-guide/overview.html">
+ Comet Overview
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="user-guide/installation.html">
+ Installing Comet
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="user-guide/expressions.html">
Supported Expressions
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="user-guide/operators.html">
+ Supported Operators
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="user-guide/datatypes.html">
+ Supported Data Types
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="user-guide/compatibility.html">
Compatibility Guide
diff --git a/searchindex.js b/searchindex.js
index d7e0fe5..983843b 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"ANSI mode": [[4, "ansi-mode"]], "ASF Links":
[[3, null]], "Additional Info": [[1, "additional-info"]], "After your debugging
is done,": [[1, "after-your-debugging-is-done"]], "Apache DataFusion Comet":
[[3, "apache-datafusion-comet"]], "Asking for Help": [[0, "asking-for-help"]],
"Benchmark": [[2, "benchmark"]], "Build & Test": [[2, "build-test"]], "CLion":
[[2, "clion"]], "Cast": [[4, "cast"]], "Cast from String to Timestamp": [[4,
"cast-from-string-to-ti [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"ANSI mode": [[4, "ansi-mode"]], "ASF Links":
[[3, null]], "Additional Info": [[1, "additional-info"]], "After your debugging
is done,": [[1, "after-your-debugging-is-done"]], "Apache DataFusion Comet":
[[3, "apache-datafusion-comet"]], "Architecture": [[9, "architecture"]],
"Asking for Help": [[0, "asking-for-help"]], "Benchmark": [[2, "benchmark"]],
"Build & Test": [[2, "build-test"]], "Building From Source": [[7,
"building-from-source"]], "CLion": [[2, " [...]
\ No newline at end of file
diff --git a/user-guide/compatibility.html b/user-guide/compatibility.html
index 9a4a05c..6e8e150 100644
--- a/user-guide/compatibility.html
+++ b/user-guide/compatibility.html
@@ -54,7 +54,7 @@ under the License.
<link rel="index" title="Index" href="../genindex.html" />
<link rel="search" title="Search" href="../search.html" />
<link rel="next" title="Contributing to Apache DataFusion Comet"
href="../contributor-guide/contributing.html" />
- <link rel="prev" title="Supported Spark Expressions"
href="expressions.html" />
+ <link rel="prev" title="Supported Spark Data Types" href="datatypes.html"
/>
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="docsearch:language" content="en">
@@ -108,11 +108,31 @@ under the License.
</span>
</p>
<ul class="current nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="overview.html">
+ Comet Overview
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="installation.html">
+ Installing Comet
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="expressions.html">
Supported Expressions
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="operators.html">
+ Supported Operators
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="datatypes.html">
+ Supported Data Types
+ </a>
+ </li>
<li class="toctree-l1 current active">
<a class="current reference internal" href="#">
Compatibility Guide
@@ -306,11 +326,11 @@ issues, and can be enabled by setting <code
class="docutils literal notranslate"
<!-- Previous / next buttons -->
<div class='prev-next-area'>
- <a class='left-prev' id="prev-link" href="expressions.html"
title="previous page">
+ <a class='left-prev' id="prev-link" href="datatypes.html" title="previous
page">
<i class="fas fa-angle-left"></i>
<div class="prev-next-info">
<p class="prev-next-subtitle">previous</p>
- <p class="prev-next-title">Supported Spark Expressions</p>
+ <p class="prev-next-title">Supported Spark Data Types</p>
</div>
</a>
<a class='right-next' id="next-link"
href="../contributor-guide/contributing.html" title="next page">
diff --git a/user-guide/expressions.html b/user-guide/datatypes.html
similarity index 77%
copy from user-guide/expressions.html
copy to user-guide/datatypes.html
index b6d8f51..43f4864 100644
--- a/user-guide/expressions.html
+++ b/user-guide/datatypes.html
@@ -24,7 +24,7 @@ under the License.
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0"
/><meta name="viewport" content="width=device-width, initial-scale=1" />
- <title>Supported Spark Expressions — Apache DataFusion Comet
documentation</title>
+ <title>Supported Spark Data Types — Apache DataFusion Comet
documentation</title>
<link href="../_static/styles/theme.css?digest=1999514e3f237ded88cf"
rel="stylesheet">
<link
href="../_static/styles/pydata-sphinx-theme.css?digest=1999514e3f237ded88cf"
rel="stylesheet">
@@ -54,7 +54,7 @@ under the License.
<link rel="index" title="Index" href="../genindex.html" />
<link rel="search" title="Search" href="../search.html" />
<link rel="next" title="Compatibility Guide" href="compatibility.html" />
- <link rel="prev" title="Apache DataFusion Comet" href="../index.html" />
+ <link rel="prev" title="Supported Spark Operators" href="operators.html" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="docsearch:language" content="en">
@@ -108,9 +108,29 @@ under the License.
</span>
</p>
<ul class="current nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="overview.html">
+ Comet Overview
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="installation.html">
+ Installing Comet
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="expressions.html">
+ Supported Expressions
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="operators.html">
+ Supported Operators
+ </a>
+ </li>
<li class="toctree-l1 current active">
<a class="current reference internal" href="#">
- Supported Expressions
+ Supported Data Types
</a>
</li>
<li class="toctree-l1">
@@ -215,7 +235,7 @@ under the License.
<div class="tocsection editthispage">
- <a
href="https://github.com/apache/datafusion-comet/edit/main/docs/source/user-guide/expressions.md";>
+ <a
href="https://github.com/apache/datafusion-comet/edit/main/docs/source/user-guide/datatypes.md";>
<i class="fas fa-pencil-alt"></i> Edit this page
</a>
</div>
@@ -252,120 +272,33 @@ under the License.
specific language governing permissions and limitations
under the License.
-->
-<section id="supported-spark-expressions">
-<h1>Supported Spark Expressions<a class="headerlink"
href="#supported-spark-expressions" title="Link to this heading">¶</a></h1>
-<p>The following Spark expressions are currently available:</p>
+<section id="supported-spark-data-types">
+<h1>Supported Spark Data Types<a class="headerlink"
href="#supported-spark-data-types" title="Link to this heading">¶</a></h1>
+<p>The following Spark data types are currently available:</p>
+<!-- based on org.apache.comet.serde.QueryPlanSerde.supportedDataType -->
<ul class="simple">
-<li><p>Literals</p></li>
-<li><p>Arithmetic Operators</p>
-<ul>
-<li><p>UnaryMinus</p></li>
-<li><p>Add/Minus/Multiply/Divide/Remainder</p></li>
-</ul>
-</li>
-<li><p>Conditional functions</p>
-<ul>
-<li><p>Case When</p></li>
-<li><p>If</p></li>
-</ul>
-</li>
-<li><p>Cast</p></li>
-<li><p>Coalesce</p></li>
-<li><p>BloomFilterMightContain</p></li>
-<li><p>Boolean functions</p>
-<ul>
-<li><p>And</p></li>
-<li><p>Or</p></li>
-<li><p>Not</p></li>
-<li><p>EqualTo</p></li>
-<li><p>EqualNullSafe</p></li>
-<li><p>GreaterThan</p></li>
-<li><p>GreaterThanOrEqual</p></li>
-<li><p>LessThan</p></li>
-<li><p>LessThanOrEqual</p></li>
-<li><p>IsNull</p></li>
-<li><p>IsNotNull</p></li>
-<li><p>In</p></li>
-</ul>
-</li>
-<li><p>String functions</p>
-<ul>
-<li><p>Substring</p></li>
-<li><p>Coalesce</p></li>
-<li><p>StringSpace</p></li>
-<li><p>Like</p></li>
-<li><p>Contains</p></li>
-<li><p>Startswith</p></li>
-<li><p>Endswith</p></li>
-<li><p>Ascii</p></li>
-<li><p>Bit_length</p></li>
-<li><p>Octet_length</p></li>
-<li><p>Upper</p></li>
-<li><p>Lower</p></li>
-<li><p>Chr</p></li>
-<li><p>Initcap</p></li>
-<li><p>Trim/Btrim/Ltrim/Rtrim</p></li>
-<li><p>Concat_ws</p></li>
-<li><p>Repeat</p></li>
-<li><p>Length</p></li>
-<li><p>Reverse</p></li>
-<li><p>Instr</p></li>
-<li><p>Replace</p></li>
-<li><p>Translate</p></li>
-</ul>
-</li>
-<li><p>Bitwise functions</p>
-<ul>
-<li><p>Shiftright/Shiftleft</p></li>
-</ul>
-</li>
-<li><p>Date/Time functions</p>
-<ul>
-<li><p>Year/Hour/Minute/Second</p></li>
-</ul>
-</li>
-<li><p>Math functions</p>
+<li><p>Primitives</p>
<ul>
-<li><p>Abs</p></li>
-<li><p>Acos</p></li>
-<li><p>Asin</p></li>
-<li><p>Atan</p></li>
-<li><p>Atan2</p></li>
-<li><p>Cos</p></li>
-<li><p>Exp</p></li>
-<li><p>Ln</p></li>
-<li><p>Log10</p></li>
-<li><p>Log2</p></li>
-<li><p>Pow</p></li>
-<li><p>Round</p></li>
-<li><p>Signum</p></li>
-<li><p>Sin</p></li>
-<li><p>Sqrt</p></li>
-<li><p>Tan</p></li>
-<li><p>Ceil</p></li>
-<li><p>Floor</p></li>
+<li><p>Boolean</p></li>
+<li><p>Byte</p></li>
+<li><p>Short</p></li>
+<li><p>Integer</p></li>
+<li><p>Long</p></li>
+<li><p>Float</p></li>
+<li><p>Double</p></li>
</ul>
</li>
-<li><p>Aggregate functions</p>
+<li><p>String</p></li>
+<li><p>Binary</p></li>
+<li><p>Decimal</p></li>
+<li><p>Temporal</p>
<ul>
-<li><p>Count</p></li>
-<li><p>Sum</p></li>
-<li><p>Max</p></li>
-<li><p>Min</p></li>
-<li><p>Avg</p></li>
-<li><p>First</p></li>
-<li><p>Last</p></li>
-<li><p>BitAnd</p></li>
-<li><p>BitOr</p></li>
-<li><p>BitXor</p></li>
-<li><p>BoolAnd</p></li>
-<li><p>BoolOr</p></li>
-<li><p>CovPopulation</p></li>
-<li><p>CovSample</p></li>
-<li><p>VariancePop</p></li>
-<li><p>VarianceSamp</p></li>
+<li><p>Date</p></li>
+<li><p>Timestamp</p></li>
+<li><p>TimestampNTZ</p></li>
</ul>
</li>
+<li><p>Null</p></li>
</ul>
</section>
@@ -375,11 +308,11 @@ under the License.
<!-- Previous / next buttons -->
<div class='prev-next-area'>
- <a class='left-prev' id="prev-link" href="../index.html" title="previous
page">
+ <a class='left-prev' id="prev-link" href="operators.html" title="previous
page">
<i class="fas fa-angle-left"></i>
<div class="prev-next-info">
<p class="prev-next-subtitle">previous</p>
- <p class="prev-next-title">Apache DataFusion Comet</p>
+ <p class="prev-next-title">Supported Spark Operators</p>
</div>
</a>
<a class='right-next' id="next-link" href="compatibility.html" title="next
page">
diff --git a/user-guide/expressions.html b/user-guide/expressions.html
index b6d8f51..51a6f7d 100644
--- a/user-guide/expressions.html
+++ b/user-guide/expressions.html
@@ -53,8 +53,8 @@ under the License.
<script async="true" defer="true"
src="https://buttons.github.io/buttons.js";></script>
<link rel="index" title="Index" href="../genindex.html" />
<link rel="search" title="Search" href="../search.html" />
- <link rel="next" title="Compatibility Guide" href="compatibility.html" />
- <link rel="prev" title="Apache DataFusion Comet" href="../index.html" />
+ <link rel="next" title="Supported Spark Operators" href="operators.html" />
+ <link rel="prev" title="Installing DataFusion Comet"
href="installation.html" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="docsearch:language" content="en">
@@ -108,11 +108,31 @@ under the License.
</span>
</p>
<ul class="current nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="overview.html">
+ Comet Overview
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="installation.html">
+ Installing Comet
+ </a>
+ </li>
<li class="toctree-l1 current active">
<a class="current reference internal" href="#">
Supported Expressions
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="operators.html">
+ Supported Operators
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="datatypes.html">
+ Supported Data Types
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="compatibility.html">
Compatibility Guide
@@ -375,17 +395,17 @@ under the License.
<!-- Previous / next buttons -->
<div class='prev-next-area'>
- <a class='left-prev' id="prev-link" href="../index.html" title="previous
page">
+ <a class='left-prev' id="prev-link" href="installation.html"
title="previous page">
<i class="fas fa-angle-left"></i>
<div class="prev-next-info">
<p class="prev-next-subtitle">previous</p>
- <p class="prev-next-title">Apache DataFusion Comet</p>
+ <p class="prev-next-title">Installing DataFusion Comet</p>
</div>
</a>
- <a class='right-next' id="next-link" href="compatibility.html" title="next
page">
+ <a class='right-next' id="next-link" href="operators.html" title="next
page">
<div class="prev-next-info">
<p class="prev-next-subtitle">next</p>
- <p class="prev-next-title">Compatibility Guide</p>
+ <p class="prev-next-title">Supported Spark Operators</p>
</div>
<i class="fas fa-angle-right"></i>
</a>
diff --git a/user-guide/installation.html b/user-guide/installation.html
new file mode 100644
index 0000000..58edda9
--- /dev/null
+++ b/user-guide/installation.html
@@ -0,0 +1,469 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+<!DOCTYPE html>
+
+<html lang="en" data-content_root="../">
+ <head>
+ <meta charset="utf-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0"
/><meta name="viewport" content="width=device-width, initial-scale=1" />
+
+ <title>Installing DataFusion Comet — Apache DataFusion Comet
documentation</title>
+
+ <link href="../_static/styles/theme.css?digest=1999514e3f237ded88cf"
rel="stylesheet">
+<link
href="../_static/styles/pydata-sphinx-theme.css?digest=1999514e3f237ded88cf"
rel="stylesheet">
+
+
+ <link rel="stylesheet"
+ href="../_static/vendor/fontawesome/5.13.0/css/all.min.css">
+ <link rel="preload" as="font" type="font/woff2" crossorigin
+ href="../_static/vendor/fontawesome/5.13.0/webfonts/fa-solid-900.woff2">
+ <link rel="preload" as="font" type="font/woff2" crossorigin
+ href="../_static/vendor/fontawesome/5.13.0/webfonts/fa-brands-400.woff2">
+
+
+
+
+
+ <link rel="stylesheet" type="text/css"
href="../_static/pygments.css?v=a746c00c" />
+ <link rel="stylesheet" type="text/css"
href="../_static/styles/pydata-sphinx-theme.css?v=1140d252" />
+ <link rel="stylesheet" type="text/css"
href="../_static/theme_overrides.css?v=c6d785ac" />
+
+ <link rel="preload" as="script"
href="../_static/scripts/pydata-sphinx-theme.js?digest=1999514e3f237ded88cf">
+
+ <script src="../_static/documentation_options.js?v=8a448e45"></script>
+ <script src="../_static/doctools.js?v=9a2dae69"></script>
+ <script src="../_static/sphinx_highlight.js?v=dc90522c"></script>
+ <script async="true" defer="true"
src="https://buttons.github.io/buttons.js";></script>
+ <link rel="index" title="Index" href="../genindex.html" />
+ <link rel="search" title="Search" href="../search.html" />
+ <link rel="next" title="Supported Spark Expressions"
href="expressions.html" />
+ <link rel="prev" title="Comet Overview" href="overview.html" />
+ <meta name="viewport" content="width=device-width, initial-scale=1" />
+ <meta name="docsearch:language" content="en">
+
+
+ <!-- Google Analytics -->
+
+ </head>
+ <body data-spy="scroll" data-target="#bd-toc-nav" data-offset="80">
+
+ <div class="container-fluid" id="banner"></div>
+
+
+
+
+ <div class="container-xl">
+ <div class="row">
+
+
+ <!-- Only show if we have sidebars configured, else just a small
margin -->
+ <div class="col-12 col-md-3 bd-sidebar">
+ <div class="sidebar-start-items"><!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+<form class="bd-search d-flex align-items-center" action="../search.html"
method="get">
+ <i class="icon fas fa-search"></i>
+ <input type="search" class="form-control" name="q" id="search-input"
placeholder="Search the docs ..." aria-label="Search the docs ..."
autocomplete="off" >
+</form>
+
+<nav class="bd-links" id="bd-docs-nav" aria-label="Main navigation">
+ <div class="bd-toc-item active">
+
+ <p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ User Guide
+ </span>
+</p>
+<ul class="current nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="overview.html">
+ Comet Overview
+ </a>
+ </li>
+ <li class="toctree-l1 current active">
+ <a class="current reference internal" href="#">
+ Installing Comet
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="expressions.html">
+ Supported Expressions
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="operators.html">
+ Supported Operators
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="datatypes.html">
+ Supported Data Types
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="compatibility.html">
+ Compatibility Guide
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ Contributor Guide
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="../contributor-guide/contributing.html">
+ Getting Started
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external"
href="https://github.com/apache/datafusion-comet";>
+ Github and Issue Tracker
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../contributor-guide/development.html">
+ Comet Development Guide
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../contributor-guide/debugging.html">
+ Comet Debugging Guide
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ ASF Links
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference external" href="https://apache.org";>
+ Apache Software Foundation
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://www.apache.org/licenses/";>
+ License
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external"
href="https://www.apache.org/foundation/sponsorship.html";>
+ Donate
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external"
href="https://www.apache.org/foundation/thanks.html";>
+ Thanks
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://www.apache.org/security/";>
+ Security
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external"
href="https://www.apache.org/foundation/policies/conduct.html";>
+ Code of conduct
+ </a>
+ </li>
+</ul>
+
+
+ </div>
+
+ <a class="navbar-brand" href="../index.html">
+ <img src="../_static/images/2x_bgwhite_original.png" class="logo"
alt="logo">
+ </a>
+</nav>
+
+ </div>
+ <div class="sidebar-end-items">
+ </div>
+ </div>
+
+
+
+
+ <div class="d-none d-xl-block col-xl-2 bd-toc">
+
+
+ <div class="toc-item">
+
+<div class="tocsection onthispage pt-5 pb-3">
+ <i class="fas fa-list"></i> On this page
+</div>
+
+<nav id="bd-toc-nav">
+ <ul class="visible nav section-nav flex-column">
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#supported-platforms">
+ Supported Platforms
+ </a>
+ </li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#requirements">
+ Requirements
+ </a>
+ </li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#using-a-published-release">
+ Using a Published Release
+ </a>
+ </li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#building-from-source">
+ Building From Source
+ </a>
+ </li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#run-spark-with-comet-enabled">
+ Run Spark with Comet enabled
+ </a>
+ <ul class="nav section-nav flex-column">
+ <li class="toc-h3 nav-item toc-entry">
+ <a class="reference internal nav-link"
href="#verify-comet-enabled-for-spark-sql-query">
+ Verify Comet enabled for Spark SQL query
+ </a>
+ </li>
+ <li class="toc-h3 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#enable-comet-shuffle">
+ Enable Comet shuffle
+ </a>
+ </li>
+ </ul>
+ </li>
+</ul>
+
+</nav>
+ </div>
+
+ <div class="toc-item">
+
+
+<div class="tocsection editthispage">
+ <a
href="https://github.com/apache/datafusion-comet/edit/main/docs/source/user-guide/installation.md";>
+ <i class="fas fa-pencil-alt"></i> Edit this page
+ </a>
+</div>
+
+ </div>
+
+
+ </div>
+
+
+
+
+
+
+ <main class="col-12 col-md-9 col-xl-7 py-md-5 pl-md-5 pr-md-4
bd-content" role="main">
+
+ <div>
+
+ <!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+<section id="installing-datafusion-comet">
+<h1>Installing DataFusion Comet<a class="headerlink"
href="#installing-datafusion-comet" title="Link to this heading">¶</a></h1>
+<p>Make sure the following requirements are met and software installed on your
machine.</p>
+<section id="supported-platforms">
+<h2>Supported Platforms<a class="headerlink" href="#supported-platforms"
title="Link to this heading">¶</a></h2>
+<ul class="simple">
+<li><p>Linux</p></li>
+<li><p>Apple OSX (Intel and Apple Silicon)</p></li>
+</ul>
+</section>
+<section id="requirements">
+<h2>Requirements<a class="headerlink" href="#requirements" title="Link to this
heading">¶</a></h2>
+<ul class="simple">
+<li><p>Apache Spark 3.2, 3.3, or 3.4</p></li>
+<li><p>JDK 8 and up</p></li>
+<li><p>GLIBC 2.17 (Centos 7) and up</p></li>
+</ul>
+</section>
+<section id="using-a-published-release">
+<h2>Using a Published Release<a class="headerlink"
href="#using-a-published-release" title="Link to this heading">¶</a></h2>
+<p>There are no public releases available yet, so it is necessary to build
from source as described in the next section.</p>
+</section>
+<section id="building-from-source">
+<h2>Building From Source<a class="headerlink" href="#building-from-source"
title="Link to this heading">¶</a></h2>
+<p>Clone the repository:</p>
+<div class="highlight-commandline notranslate"><div
class="highlight"><pre><span></span>git clone
https://github.com/apache/datafusion-comet.git
+</pre></div>
+</div>
+<p>Build Comet for a specific Spark version:</p>
+<div class="highlight-commandline notranslate"><div
class="highlight"><pre><span></span>cd datafusion-comet
+make release PROFILES="-Pspark-3.4"
+</pre></div>
+</div>
+<p>Note that the project builds for Scala 2.12 by default but can be built for
Scala 2.13 using an additional profile:</p>
+<div class="highlight-commandline notranslate"><div
class="highlight"><pre><span></span>make release PROFILES="-Pspark-3.4
-Pscala-2.13"
+</pre></div>
+</div>
+</section>
+<section id="run-spark-with-comet-enabled">
+<h2>Run Spark with Comet enabled<a class="headerlink"
href="#run-spark-with-comet-enabled" title="Link to this heading">¶</a></h2>
+<p>Make sure <code class="docutils literal notranslate"><span
class="pre">SPARK_HOME</span></code> points to the same Spark version as Comet
was built for.</p>
+<div class="highlight-commandline notranslate"><div
class="highlight"><pre><span></span>$SPARK_HOME/bin/spark-shell \
+ --jars spark/target/comet-spark-spark3.4_2.12-0.1.0-SNAPSHOT.jar \
+ --conf spark.sql.extensions=org.apache.comet.CometSparkSessionExtensions \
+ --conf spark.comet.enabled=true \
+ --conf spark.comet.exec.enabled=true \
+ --conf spark.comet.exec.all.enabled=true
+</pre></div>
+</div>
+<section id="verify-comet-enabled-for-spark-sql-query">
+<h3>Verify Comet enabled for Spark SQL query<a class="headerlink"
href="#verify-comet-enabled-for-spark-sql-query" title="Link to this
heading">¶</a></h3>
+<p>Create a test Parquet source</p>
+<div class="highlight-scala notranslate"><div
class="highlight"><pre><span></span><span class="n">scala</span><span
class="o">></span><span class="w"> </span><span class="p">(</span><span
class="mi">0</span><span class="w"> </span><span class="n">until</span><span
class="w"> </span><span class="mi">10</span><span class="p">).</span><span
class="n">toDF</span><span class="p">(</span><span
class="s">"a"</span><span class="p">).</span><span
class="n">write</span><span class="p" [...]
+</pre></div>
+</div>
+<p>Query the data from the test source and check:</p>
+<ul class="simple">
+<li><p>INFO message shows the native Comet library has been
initialized.</p></li>
+<li><p>The query plan reflects Comet operators being used for this query
instead of Spark ones</p></li>
+</ul>
+<div class="highlight-scala notranslate"><div
class="highlight"><pre><span></span><span class="n">scala</span><span
class="o">></span><span class="w"> </span><span class="n">spark</span><span
class="p">.</span><span class="n">read</span><span class="p">.</span><span
class="n">parquet</span><span class="p">(</span><span
class="s">"/tmp/test"</span><span class="p">).</span><span
class="n">createOrReplaceTempView</span><span class="p">(</span><span
class="s">"t1"</spa [...]
+<span class="n">scala</span><span class="o">></span><span class="w">
</span><span class="n">spark</span><span class="p">.</span><span
class="n">sql</span><span class="p">(</span><span class="s">"select * from
t1 where a > 5"</span><span class="p">).</span><span
class="n">explain</span>
+<span class="nc">INFO</span><span class="w"> </span><span
class="n">src</span><span class="o">/</span><span class="n">lib</span><span
class="p">.</span><span class="n">rs</span><span class="p">:</span><span
class="w"> </span><span class="nc">Comet</span><span class="w"> </span><span
class="n">native</span><span class="w"> </span><span
class="n">library</span><span class="w"> </span><span
class="n">initialized</span>
+<span class="o">==</span><span class="w"> </span><span
class="nc">Physical</span><span class="w"> </span><span
class="nc">Plan</span><span class="w"> </span><span class="o">==</span>
+<span class="w"> </span><span class="o">*</span><span
class="p">(</span><span class="mi">1</span><span class="p">)</span><span
class="w"> </span><span class="nc">ColumnarToRow</span>
+<span class="w"> </span><span class="o">+-</span><span class="w">
</span><span class="nc">CometFilter</span><span class="w"> </span><span
class="p">[</span><span class="n">a#</span><span class="mi">14</span><span
class="p">],</span><span class="w"> </span><span class="p">(</span><span
class="n">isnotnull</span><span class="p">(</span><span
class="n">a#</span><span class="mi">14</span><span class="p">)</span><span
class="w"> </span><span class="nc">AND</span><span class="w"> </span [...]
+<span class="w"> </span><span class="o">+-</span><span class="w">
</span><span class="nc">CometScan</span><span class="w"> </span><span
class="n">parquet</span><span class="w"> </span><span class="p">[</span><span
class="n">a#</span><span class="mi">14</span><span class="p">]</span><span
class="w"> </span><span class="nc">Batched</span><span class="p">:</span><span
class="w"> </span><span class="kc">true</span><span class="p">,</span><span
class="w"> </span><span class="nc">Data [...]
+<span class="w"> </span><span class="nc">Format</span><span
class="p">:</span><span class="w"> </span><span
class="nc">CometParquet</span><span class="p">,</span><span class="w">
</span><span class="nc">Location</span><span class="p">:</span><span class="w">
</span><span class="nc">InMemoryFileIndex</span><span class="p">(</span><span
class="mi">1</span><span class="w"> </span><span class="n">paths</span><span
class="p">)[</span><span class="n">file</span><span class="p">:</s [...]
+<span class="w"> </span><span class="nc">PushedFilters</span><span
class="p">:</span><span class="w"> </span><span class="p">[</span><span
class="nc">IsNotNull</span><span class="p">(</span><span
class="n">a</span><span class="p">),</span><span class="w"> </span><span
class="nc">GreaterThan</span><span class="p">(</span><span
class="n">a</span><span class="p">,</span><span class="mi">5</span><span
class="p">)],</span><span class="w"> </span><span class="nc">ReadSchema</span><
[...]
+</pre></div>
+</div>
+</section>
+<section id="enable-comet-shuffle">
+<h3>Enable Comet shuffle<a class="headerlink" href="#enable-comet-shuffle"
title="Link to this heading">¶</a></h3>
+<p>Comet shuffle feature is disabled by default. To enable it, please add
related configs:</p>
+<div class="highlight-default notranslate"><div
class="highlight"><pre><span></span><span class="o">--</span><span
class="n">conf</span> <span class="n">spark</span><span class="o">.</span><span
class="n">shuffle</span><span class="o">.</span><span
class="n">manager</span><span class="o">=</span><span class="n">org</span><span
class="o">.</span><span class="n">apache</span><span class="o">.</span><span
class="n">spark</span><span class="o">.</span><span class="n">sql</span><span
class="o [...]
+<span class="o">--</span><span class="n">conf</span> <span
class="n">spark</span><span class="o">.</span><span class="n">comet</span><span
class="o">.</span><span class="n">exec</span><span class="o">.</span><span
class="n">shuffle</span><span class="o">.</span><span
class="n">enabled</span><span class="o">=</span><span class="n">true</span>
+</pre></div>
+</div>
+<p>Above configs enable Comet native shuffle which only supports hash
partition and single partition.
+Comet native shuffle doesn’t support complex types yet.</p>
+<p>Comet doesn’t have official release yet so currently the only way to test
it is to build jar and include it in your
+Spark application. Depending on your deployment mode you may also need to set
the driver & executor class path(s) to
+explicitly contain Comet otherwise Spark may use a different class-loader for
the Comet components than its internal
+components which will then fail at runtime. For example:</p>
+<div class="highlight-default notranslate"><div
class="highlight"><pre><span></span><span class="o">--</span><span
class="n">driver</span><span class="o">-</span><span
class="n">class</span><span class="o">-</span><span class="n">path</span> <span
class="n">spark</span><span class="o">/</span><span
class="n">target</span><span class="o">/</span><span
class="n">comet</span><span class="o">-</span><span class="n">spark</span><span
class="o">-</span><span class="n">spark3</span><span class= [...]
+</pre></div>
+</div>
+<p>Some cluster managers may require additional configuration, see
https://spark.apache.org/docs/latest/cluster-overview.html</p>
+<p>To enable columnar shuffle which supports all partitioning and basic
complex types, one more config is required:</p>
+<div class="highlight-default notranslate"><div
class="highlight"><pre><span></span><span class="o">--</span><span
class="n">conf</span> <span class="n">spark</span><span class="o">.</span><span
class="n">comet</span><span class="o">.</span><span
class="n">columnar</span><span class="o">.</span><span
class="n">shuffle</span><span class="o">.</span><span
class="n">enabled</span><span class="o">=</span><span class="n">true</span>
+</pre></div>
+</div>
+</section>
+</section>
+</section>
+
+
+ </div>
+
+
+ <!-- Previous / next buttons -->
+<div class='prev-next-area'>
+ <a class='left-prev' id="prev-link" href="overview.html" title="previous
page">
+ <i class="fas fa-angle-left"></i>
+ <div class="prev-next-info">
+ <p class="prev-next-subtitle">previous</p>
+ <p class="prev-next-title">Comet Overview</p>
+ </div>
+ </a>
+ <a class='right-next' id="next-link" href="expressions.html" title="next
page">
+ <div class="prev-next-info">
+ <p class="prev-next-subtitle">next</p>
+ <p class="prev-next-title">Supported Spark Expressions</p>
+ </div>
+ <i class="fas fa-angle-right"></i>
+ </a>
+</div>
+
+ </main>
+
+
+ </div>
+ </div>
+
+ <script
src="../_static/scripts/pydata-sphinx-theme.js?digest=1999514e3f237ded88cf"></script>
+
+<!-- Based on pydata_sphinx_theme/footer.html -->
+<footer class="footer mt-5 mt-md-0">
+ <div class="container">
+
+ <div class="footer-item">
+ <p class="copyright">
+ © Copyright 2023-2024, Apache Software Foundation.<br>
+</p>
+ </div>
+
+ <div class="footer-item">
+ <p class="sphinx-version">
+Created using <a href="http://sphinx-doc.org/";>Sphinx</a> 7.3.7.<br>
+</p>
+ </div>
+
+ <div class="footer-item">
+ <p>Apache DataFusion, Apache DataFusion Comet, Apache, the Apache
feather logo, and the Apache DataFusion project logo</p>
+ <p>are either registered trademarks or trademarks of The Apache Software
Foundation in the United States and other countries.</p>
+ </div>
+ </div>
+</footer>
+
+
+ </body>
+</html>
\ No newline at end of file
diff --git a/user-guide/expressions.html b/user-guide/operators.html
similarity index 75%
copy from user-guide/expressions.html
copy to user-guide/operators.html
index b6d8f51..3413b78 100644
--- a/user-guide/expressions.html
+++ b/user-guide/operators.html
@@ -24,7 +24,7 @@ under the License.
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0"
/><meta name="viewport" content="width=device-width, initial-scale=1" />
- <title>Supported Spark Expressions — Apache DataFusion Comet
documentation</title>
+ <title>Supported Spark Operators — Apache DataFusion Comet
documentation</title>
<link href="../_static/styles/theme.css?digest=1999514e3f237ded88cf"
rel="stylesheet">
<link
href="../_static/styles/pydata-sphinx-theme.css?digest=1999514e3f237ded88cf"
rel="stylesheet">
@@ -53,8 +53,8 @@ under the License.
<script async="true" defer="true"
src="https://buttons.github.io/buttons.js";></script>
<link rel="index" title="Index" href="../genindex.html" />
<link rel="search" title="Search" href="../search.html" />
- <link rel="next" title="Compatibility Guide" href="compatibility.html" />
- <link rel="prev" title="Apache DataFusion Comet" href="../index.html" />
+ <link rel="next" title="Supported Spark Data Types" href="datatypes.html"
/>
+ <link rel="prev" title="Supported Spark Expressions"
href="expressions.html" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="docsearch:language" content="en">
@@ -108,9 +108,29 @@ under the License.
</span>
</p>
<ul class="current nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="overview.html">
+ Comet Overview
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="installation.html">
+ Installing Comet
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="expressions.html">
+ Supported Expressions
+ </a>
+ </li>
<li class="toctree-l1 current active">
<a class="current reference internal" href="#">
- Supported Expressions
+ Supported Operators
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="datatypes.html">
+ Supported Data Types
</a>
</li>
<li class="toctree-l1">
@@ -215,7 +235,7 @@ under the License.
<div class="tocsection editthispage">
- <a
href="https://github.com/apache/datafusion-comet/edit/main/docs/source/user-guide/expressions.md";>
+ <a
href="https://github.com/apache/datafusion-comet/edit/main/docs/source/user-guide/operators.md";>
<i class="fas fa-pencil-alt"></i> Edit this page
</a>
</div>
@@ -252,120 +272,20 @@ under the License.
specific language governing permissions and limitations
under the License.
-->
-<section id="supported-spark-expressions">
-<h1>Supported Spark Expressions<a class="headerlink"
href="#supported-spark-expressions" title="Link to this heading">¶</a></h1>
-<p>The following Spark expressions are currently available:</p>
+<section id="supported-spark-operators">
+<h1>Supported Spark Operators<a class="headerlink"
href="#supported-spark-operators" title="Link to this heading">¶</a></h1>
+<p>The following Spark operators are currently available:</p>
<ul class="simple">
-<li><p>Literals</p></li>
-<li><p>Arithmetic Operators</p>
-<ul>
-<li><p>UnaryMinus</p></li>
-<li><p>Add/Minus/Multiply/Divide/Remainder</p></li>
-</ul>
-</li>
-<li><p>Conditional functions</p>
-<ul>
-<li><p>Case When</p></li>
-<li><p>If</p></li>
-</ul>
-</li>
-<li><p>Cast</p></li>
-<li><p>Coalesce</p></li>
-<li><p>BloomFilterMightContain</p></li>
-<li><p>Boolean functions</p>
-<ul>
-<li><p>And</p></li>
-<li><p>Or</p></li>
-<li><p>Not</p></li>
-<li><p>EqualTo</p></li>
-<li><p>EqualNullSafe</p></li>
-<li><p>GreaterThan</p></li>
-<li><p>GreaterThanOrEqual</p></li>
-<li><p>LessThan</p></li>
-<li><p>LessThanOrEqual</p></li>
-<li><p>IsNull</p></li>
-<li><p>IsNotNull</p></li>
-<li><p>In</p></li>
-</ul>
-</li>
-<li><p>String functions</p>
-<ul>
-<li><p>Substring</p></li>
-<li><p>Coalesce</p></li>
-<li><p>StringSpace</p></li>
-<li><p>Like</p></li>
-<li><p>Contains</p></li>
-<li><p>Startswith</p></li>
-<li><p>Endswith</p></li>
-<li><p>Ascii</p></li>
-<li><p>Bit_length</p></li>
-<li><p>Octet_length</p></li>
-<li><p>Upper</p></li>
-<li><p>Lower</p></li>
-<li><p>Chr</p></li>
-<li><p>Initcap</p></li>
-<li><p>Trim/Btrim/Ltrim/Rtrim</p></li>
-<li><p>Concat_ws</p></li>
-<li><p>Repeat</p></li>
-<li><p>Length</p></li>
-<li><p>Reverse</p></li>
-<li><p>Instr</p></li>
-<li><p>Replace</p></li>
-<li><p>Translate</p></li>
-</ul>
-</li>
-<li><p>Bitwise functions</p>
-<ul>
-<li><p>Shiftright/Shiftleft</p></li>
-</ul>
-</li>
-<li><p>Date/Time functions</p>
-<ul>
-<li><p>Year/Hour/Minute/Second</p></li>
-</ul>
-</li>
-<li><p>Math functions</p>
-<ul>
-<li><p>Abs</p></li>
-<li><p>Acos</p></li>
-<li><p>Asin</p></li>
-<li><p>Atan</p></li>
-<li><p>Atan2</p></li>
-<li><p>Cos</p></li>
-<li><p>Exp</p></li>
-<li><p>Ln</p></li>
-<li><p>Log10</p></li>
-<li><p>Log2</p></li>
-<li><p>Pow</p></li>
-<li><p>Round</p></li>
-<li><p>Signum</p></li>
-<li><p>Sin</p></li>
-<li><p>Sqrt</p></li>
-<li><p>Tan</p></li>
-<li><p>Ceil</p></li>
-<li><p>Floor</p></li>
-</ul>
-</li>
-<li><p>Aggregate functions</p>
-<ul>
-<li><p>Count</p></li>
-<li><p>Sum</p></li>
-<li><p>Max</p></li>
-<li><p>Min</p></li>
-<li><p>Avg</p></li>
-<li><p>First</p></li>
-<li><p>Last</p></li>
-<li><p>BitAnd</p></li>
-<li><p>BitOr</p></li>
-<li><p>BitXor</p></li>
-<li><p>BoolAnd</p></li>
-<li><p>BoolOr</p></li>
-<li><p>CovPopulation</p></li>
-<li><p>CovSample</p></li>
-<li><p>VariancePop</p></li>
-<li><p>VarianceSamp</p></li>
-</ul>
-</li>
+<li><p>FileSourceScanExec/BatchScanExec for Parquet</p></li>
+<li><p>Projection</p></li>
+<li><p>Filter</p></li>
+<li><p>Sort</p></li>
+<li><p>Hash Aggregate</p></li>
+<li><p>Limit</p></li>
+<li><p>Sort-merge Join</p></li>
+<li><p>Hash Join</p></li>
+<li><p>Shuffle</p></li>
+<li><p>Expand</p></li>
</ul>
</section>
@@ -375,17 +295,17 @@ under the License.
<!-- Previous / next buttons -->
<div class='prev-next-area'>
- <a class='left-prev' id="prev-link" href="../index.html" title="previous
page">
+ <a class='left-prev' id="prev-link" href="expressions.html"
title="previous page">
<i class="fas fa-angle-left"></i>
<div class="prev-next-info">
<p class="prev-next-subtitle">previous</p>
- <p class="prev-next-title">Apache DataFusion Comet</p>
+ <p class="prev-next-title">Supported Spark Expressions</p>
</div>
</a>
- <a class='right-next' id="next-link" href="compatibility.html" title="next
page">
+ <a class='right-next' id="next-link" href="datatypes.html" title="next
page">
<div class="prev-next-info">
<p class="prev-next-subtitle">next</p>
- <p class="prev-next-title">Compatibility Guide</p>
+ <p class="prev-next-title">Supported Spark Data Types</p>
</div>
<i class="fas fa-angle-right"></i>
</a>
diff --git a/user-guide/compatibility.html b/user-guide/overview.html
similarity index 74%
copy from user-guide/compatibility.html
copy to user-guide/overview.html
index 9a4a05c..34e15de 100644
--- a/user-guide/compatibility.html
+++ b/user-guide/overview.html
@@ -24,7 +24,7 @@ under the License.
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0"
/><meta name="viewport" content="width=device-width, initial-scale=1" />
- <title>Compatibility Guide — Apache DataFusion Comet
documentation</title>
+ <title>Comet Overview — Apache DataFusion Comet
documentation</title>
<link href="../_static/styles/theme.css?digest=1999514e3f237ded88cf"
rel="stylesheet">
<link
href="../_static/styles/pydata-sphinx-theme.css?digest=1999514e3f237ded88cf"
rel="stylesheet">
@@ -53,8 +53,8 @@ under the License.
<script async="true" defer="true"
src="https://buttons.github.io/buttons.js";></script>
<link rel="index" title="Index" href="../genindex.html" />
<link rel="search" title="Search" href="../search.html" />
- <link rel="next" title="Contributing to Apache DataFusion Comet"
href="../contributor-guide/contributing.html" />
- <link rel="prev" title="Supported Spark Expressions"
href="expressions.html" />
+ <link rel="next" title="Installing DataFusion Comet"
href="installation.html" />
+ <link rel="prev" title="Apache DataFusion Comet" href="../index.html" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="docsearch:language" content="en">
@@ -108,13 +108,33 @@ under the License.
</span>
</p>
<ul class="current nav bd-sidenav">
+ <li class="toctree-l1 current active">
+ <a class="current reference internal" href="#">
+ Comet Overview
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="installation.html">
+ Installing Comet
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference internal" href="expressions.html">
Supported Expressions
</a>
</li>
- <li class="toctree-l1 current active">
- <a class="current reference internal" href="#">
+ <li class="toctree-l1">
+ <a class="reference internal" href="operators.html">
+ Supported Operators
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="datatypes.html">
+ Supported Data Types
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="compatibility.html">
Compatibility Guide
</a>
</li>
@@ -212,21 +232,19 @@ under the License.
<nav id="bd-toc-nav">
<ul class="visible nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry">
- <a class="reference internal nav-link" href="#ansi-mode">
- ANSI mode
+ <a class="reference internal nav-link" href="#architecture">
+ Architecture
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
- <a class="reference internal nav-link" href="#cast">
- Cast
+ <a class="reference internal nav-link" href="#current-status">
+ Current Status
+ </a>
+ </li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link"
href="#feature-parity-with-apache-spark">
+ Feature Parity with Apache Spark
</a>
- <ul class="nav section-nav flex-column">
- <li class="toc-h3 nav-item toc-entry">
- <a class="reference internal nav-link"
href="#cast-from-string-to-timestamp">
- Cast from String to Timestamp
- </a>
- </li>
- </ul>
</li>
</ul>
@@ -237,7 +255,7 @@ under the License.
<div class="tocsection editthispage">
- <a
href="https://github.com/apache/datafusion-comet/edit/main/docs/source/user-guide/compatibility.md";>
+ <a
href="https://github.com/apache/datafusion-comet/edit/main/docs/source/user-guide/overview.md";>
<i class="fas fa-pencil-alt"></i> Edit this page
</a>
</div>
@@ -274,29 +292,38 @@ under the License.
specific language governing permissions and limitations
under the License.
-->
-<section id="compatibility-guide">
-<h1>Compatibility Guide<a class="headerlink" href="#compatibility-guide"
title="Link to this heading">¶</a></h1>
-<p>Comet aims to provide consistent results with the version of Apache Spark
that is being used.</p>
-<p>This guide offers information about areas of functionality where there are
known differences.</p>
-<section id="ansi-mode">
-<h2>ANSI mode<a class="headerlink" href="#ansi-mode" title="Link to this
heading">¶</a></h2>
-<p>Comet currently ignores ANSI mode in most cases, and therefore can produce
different results than Spark. By default,
-Comet will fall back to Spark if ANSI mode is enabled. To enable Comet to
accelerate queries when ANSI mode is enabled,
-specify <code class="docutils literal notranslate"><span
class="pre">spark.comet.ansi.enabled=true</span></code> in the Spark
configuration. Comet’s ANSI support is experimental and should not
-be used in production.</p>
-<p>There is an <a class="reference external"
href="https://github.com/apache/datafusion-comet/issues/313";>epic</a> where we
are tracking the work to fully implement ANSI support.</p>
+<section id="comet-overview">
+<h1>Comet Overview<a class="headerlink" href="#comet-overview" title="Link to
this heading">¶</a></h1>
+<p>Comet runs Spark SQL queries using the native Apache DataFusion runtime,
which is
+typically faster and more resource efficient than JVM based runtimes.</p>
+<p><img alt="Comet Overview" src="../_images/comet-overview.png" /></p>
+<p>Comet aims to support:</p>
+<ul class="simple">
+<li><p>a native Parquet implementation, including both reader and
writer</p></li>
+<li><p>full implementation of Spark operators, including
+Filter/Project/Aggregation/Join/Exchange etc.</p></li>
+<li><p>full implementation of Spark built-in expressions</p></li>
+<li><p>a UDF framework for users to migrate their existing UDF to
native</p></li>
+</ul>
+<section id="architecture">
+<h2>Architecture<a class="headerlink" href="#architecture" title="Link to this
heading">¶</a></h2>
+<p>The following diagram illustrates the architecture of Comet:</p>
+<p><img alt="Comet System Diagram" src="../_images/comet-system-diagram.png"
/></p>
</section>
-<section id="cast">
-<h2>Cast<a class="headerlink" href="#cast" title="Link to this
heading">¶</a></h2>
-<p>Comet currently delegates to Apache DataFusion for most cast operations,
and this means that the behavior is not
-guaranteed to be consistent with Spark.</p>
-<p>There is an <a class="reference external"
href="https://github.com/apache/datafusion-comet/issues/286";>epic</a> where we
are tracking the work to implement Spark-compatible cast expressions.</p>
-<section id="cast-from-string-to-timestamp">
-<h3>Cast from String to Timestamp<a class="headerlink"
href="#cast-from-string-to-timestamp" title="Link to this heading">¶</a></h3>
-<p>Casting from String to Timestamp is disabled by default due to
incompatibilities with Spark, including timezone
-issues, and can be enabled by setting <code class="docutils literal
notranslate"><span
class="pre">spark.comet.castStringToTimestamp=true</span></code>. See the
-<a class="reference external"
href="https://github.com/apache/datafusion-comet/issues/328";>tracking issue</a>
for more information.</p>
+<section id="current-status">
+<h2>Current Status<a class="headerlink" href="#current-status" title="Link to
this heading">¶</a></h2>
+<p>The project is currently integrated into Apache Spark 3.2, 3.3, and 3.4.</p>
</section>
+<section id="feature-parity-with-apache-spark">
+<h2>Feature Parity with Apache Spark<a class="headerlink"
href="#feature-parity-with-apache-spark" title="Link to this heading">¶</a></h2>
+<p>The project strives to keep feature parity with Apache Spark, that is,
+users should expect the same behavior (w.r.t features, configurations,
+query results, etc) with Comet turned on or turned off in their Spark
+jobs. In addition, Comet extension should automatically detect unsupported
+features and fallback to Spark engine.</p>
+<p>To achieve this, besides unit tests within Comet itself, we also re-use
+Spark SQL tests and make sure they all pass with Comet extension
+enabled.</p>
</section>
</section>
@@ -306,17 +333,17 @@ issues, and can be enabled by setting <code
class="docutils literal notranslate"
<!-- Previous / next buttons -->
<div class='prev-next-area'>
- <a class='left-prev' id="prev-link" href="expressions.html"
title="previous page">
+ <a class='left-prev' id="prev-link" href="../index.html" title="previous
page">
<i class="fas fa-angle-left"></i>
<div class="prev-next-info">
<p class="prev-next-subtitle">previous</p>
- <p class="prev-next-title">Supported Spark Expressions</p>
+ <p class="prev-next-title">Apache DataFusion Comet</p>
</div>
</a>
- <a class='right-next' id="next-link"
href="../contributor-guide/contributing.html" title="next page">
+ <a class='right-next' id="next-link" href="installation.html" title="next
page">
<div class="prev-next-info">
<p class="prev-next-subtitle">next</p>
- <p class="prev-next-title">Contributing to Apache DataFusion Comet</p>
+ <p class="prev-next-title">Installing DataFusion Comet</p>
</div>
<i class="fas fa-angle-right"></i>
</a>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
