This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-site.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 7449deb Commit build products
7449deb is described below
commit 7449debf60a53a45399a1e54bbe08299ada5b454
Author: Build Pelican (action) <[email protected]>
AuthorDate: Wed Oct 22 02:07:06 2025 +0000
Commit build products
---
.../2025/10/21/datafusion-comet-0.11.0/index.html | 263 +++++++++++++++++++++
output/author/pmc.html | 32 +++
output/category/blog.html | 32 +++
output/feed.xml | 24 +-
output/feeds/all-en.atom.xml | 149 +++++++++++-
output/feeds/blog.atom.xml | 149 +++++++++++-
output/feeds/pmc.atom.xml | 149 +++++++++++-
output/feeds/pmc.rss.xml | 24 +-
output/images/comet-0.11.0/tpch_allqueries.png | Bin 0 -> 32703 bytes
.../images/comet-0.11.0/tpch_queries_compare.png | Bin 0 -> 34547 bytes
output/index.html | 41 ++++
11 files changed, 858 insertions(+), 5 deletions(-)
diff --git a/output/2025/10/21/datafusion-comet-0.11.0/index.html
b/output/2025/10/21/datafusion-comet-0.11.0/index.html
new file mode 100644
index 0000000..62ce112
--- /dev/null
+++ b/output/2025/10/21/datafusion-comet-0.11.0/index.html
@@ -0,0 +1,263 @@
+<!doctype html>
+<html class="no-js" lang="en" dir="ltr">
+ <head>
+ <meta charset="utf-8">
+ <meta http-equiv="x-ua-compatible" content="ie=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+ <title>Apache DataFusion Comet 0.11.0 Release - Apache DataFusion
Blog</title>
+<link href="/blog/css/bootstrap.min.css" rel="stylesheet">
+<link href="/blog/css/fontawesome.all.min.css" rel="stylesheet">
+<link href="/blog/css/headerlink.css" rel="stylesheet">
+<link href="/blog/highlight/default.min.css" rel="stylesheet">
+<link href="/blog/css/app.css" rel="stylesheet">
+<script src="/blog/highlight/highlight.js"></script>
+<script>hljs.highlightAll();</script> </head>
+ <body class="d-flex flex-column h-100">
+ <main class="flex-shrink-0">
+<!-- nav bar -->
+<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth
navbar example">
+ <div class="container-fluid">
+ <a class="navbar-brand" href="/blog"><img
src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache
DataFusion Blog</a>
+ <button class="navbar-toggler" type="button" data-bs-toggle="collapse"
data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false"
aria-label="Toggle navigation">
+ <span class="navbar-toggler-icon"></span>
+ </button>
+
+ <div class="collapse navbar-collapse" id="navbarADP">
+ <ul class="navbar-nav me-auto mb-2 mb-lg-0">
+ <li class="nav-item">
+ <a class="nav-link" href="/blog/about.html">About</a>
+ </li>
+ <li class="nav-item">
+ <a class="nav-link" href="/blog/feed.xml">RSS</a>
+ </li>
+ </ul>
+ </div>
+ </div>
+</nav>
+<!-- article contents -->
+<div id="contents">
+ <div class="bg-white p-4 p-md-5 rounded">
+ <div class="row justify-content-center">
+ <div class="col-12 col-md-8 main-content">
+ <h1>
+ Apache DataFusion Comet 0.11.0 Release
+ </h1>
+ <p>Posted on: Tue 21 October 2025 by pmc</p>
+
+ <aside class="toc-container d-md-none mb-2">
+ <div class="toc"><span class="toctitle">Contents</span><ul>
+<li><a href="#release-highlights">Release Highlights</a><ul>
+<li><a href="#parquet-modular-encryption-support">Parquet Modular Encryption
Support</a></li>
+<li><a href="#improved-memory-management">Improved Memory Management</a></li>
+<li><a href="#improved-apache-spark-40-support">Improved Apache Spark 4.0
Support</a></li>
+<li><a href="#complex-types-for-columnar-shuffle">Complex Types for Columnar
Shuffle</a></li>
+<li><a href="#rangepartitioning-for-native-shuffle">RangePartitioning for
Native Shuffle</a></li>
+<li><a href="#new-functionality">New Functionality</a></li>
+<li><a href="#performance-improvements">Performance Improvements</a></li>
+<li><a href="#comet-0110-tpc-h-performance">Comet 0.11.0 TPC-H
Performance</a></li>
+<li><a href="#apache-iceberg-support">Apache Iceberg Support</a></li>
+<li><a href="#ux-improvements">UX Improvements</a></li>
+<li><a href="#bug-fixes">Bug Fixes</a></li>
+<li><a href="#documentation-updates">Documentation Updates</a></li>
+<li><a href="#spark-compatibility">Spark Compatibility</a></li>
+</ul>
+</li>
+<li><a href="#getting-involved">Getting Involved</a></li>
+</ul>
+</div>
+ </aside>
+
+ <!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<p>The Apache DataFusion PMC is pleased to announce version 0.11.0 of the <a
href="https://datafusion.apache.org/comet/";>Comet</a> subproject.</p>
+<p>Comet is an accelerator for Apache Spark that translates Spark physical
plans to DataFusion physical plans for
+improved performance and efficiency without requiring any code changes.</p>
+<p>This release covers approximately five weeks of development work and is the
result of merging 131 PRs from 15
+contributors. See the <a
href="https://github.com/apache/datafusion-comet/blob/main/dev/changelog/0.11.0.md";>change
log</a> for more information.</p>
+<h2 id="release-highlights">Release Highlights<a class="headerlink"
href="#release-highlights" title="Permanent link">¶</a></h2>
+<h3 id="parquet-modular-encryption-support">Parquet Modular Encryption
Support<a class="headerlink" href="#parquet-modular-encryption-support"
title="Permanent link">¶</a></h3>
+<p>Spark supports Parquet Modular Encryption to independently encrypt column
values and metadata. Furthermore, Spark supports custom encryption factories
for users to provide their own key-management service (KMS) implementations.
Thanks to <a href="https://github.com/apache/arrow-rs/issues/7278";>a</a> <a
href="https://github.com/apache/datafusion/issues/15216";>number</a> <a
href="https://github.com/apache/datafusion/pull/16351";>of</a> <a
href="https://github.com/apache/datafusion/pull/1 [...]
+<h3 id="improved-memory-management">Improved Memory Management<a
class="headerlink" href="#improved-memory-management" title="Permanent
link">¶</a></h3>
+<p>Comet 0.11.0 introduces significant improvements to memory management,
making it easier to deploy and more resilient to out-of-memory conditions:</p>
+<ul>
+<li><strong>Changed default memory pool</strong>: The default off-heap memory
pool has been <a
href="https://github.com/apache/datafusion-comet/pull/2526";>changed from
<code>greedy_unified</code> to <code>fair_unified</code></a>, providing better
memory fairness across operations</li>
+<li><strong>Off-heap deployment recommended</strong>: To simplify
configuration and improve performance, Comet now expects to be deployed with
Spark's off-heap memory configuration. <a
href="https://github.com/apache/datafusion-comet/pull/2554";>On-heap memory is
still available</a> for development and debugging, but is not recommended for
deployment</li>
+<li><strong>Better disk management</strong>: The <a
href="https://github.com/apache/datafusion-comet/pull/2479";>DiskManager
<code>max_temp_directory_size</code> is now configurable</a> for better control
over temporary disk usage</li>
+<li><strong>Enhanced safety</strong>: Memory pool operations now <a
href="https://github.com/apache/datafusion-comet/pull/2455";>use checked
arithmetic operations</a> to prevent overflow issues</li>
+</ul>
+<p>These changes make Comet significantly easier to configure and deploy in
production environments.</p>
+<h3 id="improved-apache-spark-40-support">Improved Apache Spark 4.0 Support<a
class="headerlink" href="#improved-apache-spark-40-support" title="Permanent
link">¶</a></h3>
+<p>Comet has improved its support for Apache Spark 4.0.1 with several
important enhancements:</p>
+<ul>
+<li><a href="https://github.com/apache/datafusion-comet/pull/2414";>Updated
support from Spark 4.0.0 to Spark 4.0.1</a></li>
+<li><a href="https://github.com/apache/datafusion-comet/pull/2514";>Spark 4.0
is now included in the release build script</a></li>
+<li>Expanded ANSI mode compatibility with several new implementations:</li>
+<li><a href="https://github.com/apache/datafusion-comet/pull/2136";>ANSI
evaluation mode arithmetic operations</a></li>
+<li><a href="https://github.com/apache/datafusion-comet/pull/2421";>ANSI mode
integral divide</a></li>
+<li><a href="https://github.com/apache/datafusion-comet/pull/2542";>ANSI mode
rounding functions</a></li>
+<li><a href="https://github.com/apache/datafusion-comet/pull/2556";>ANSI mode
remainder function</a></li>
+</ul>
+<p>Spark 4.0 compatible jar files are now available on Maven Central. See the
<a
href="https://datafusion.apache.org/comet/user-guide/0.11/installation.html#using-a-published-jar-file";>installation
guide</a> for instructions on using published jar files.</p>
+<h3 id="complex-types-for-columnar-shuffle">Complex Types for Columnar
Shuffle<a class="headerlink" href="#complex-types-for-columnar-shuffle"
title="Permanent link">¶</a></h3>
+<p><a href="https://github.com/ashdnazg";>ashdnazg</a> submitted a <a
href="https://github.com/apache/datafusion-comet/pull/2571";>fantastic
refactoring PR</a> that simplified the logic for writing rows in Comet’s
JVM-based, columnar shuffle. A benefit of this refactoring is better support
for complex types (<em>e.g.,</em> structs, lists, and arrays) in columnar
shuffle. Comet no longer falls back to Spark to shuffle these types, enabling
native acceleration for queries involving nested da [...]
+<h3 id="rangepartitioning-for-native-shuffle">RangePartitioning for Native
Shuffle<a class="headerlink" href="#rangepartitioning-for-native-shuffle"
title="Permanent link">¶</a></h3>
+<p>Comet's native shuffle now <a
href="https://github.com/apache/datafusion-comet/pull/2258";>supports
RangePartitioning</a>, providing better performance for operations that require
range-based data distribution. Comet now matches Spark behavior for computing
and distributing range boundaries, and serializes them to native execution for
faster shuffle operations.</p>
+<h3 id="new-functionality">New Functionality<a class="headerlink"
href="#new-functionality" title="Permanent link">¶</a></h3>
+<p>The following SQL functions are now supported:</p>
+<ul>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2411";><code>weekday</code></a>
- Extract day of week from date</li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2102";><code>lpad</code></a>
- Left pad a string with column support for pad length</li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2099";><code>rpad</code></a>
- Right pad a string with <a
href="https://github.com/apache/datafusion-comet/pull/2436";>column support and
additional character support</a></li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2481";><code>reverse</code></a>
- Support for ArrayType input in addition to strings</li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2429";><code>count(distinct)</code></a>
- Native support without falling back to Spark</li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2466";><code>bit_get</code></a>
- Get bit value at position</li>
+</ul>
+<p>New expression capabilities include:</p>
+<ul>
+<li><a href="https://github.com/apache/datafusion-comet/pull/2181";>Nested
array literal support</a></li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2425";>Array-to-string
cast support</a></li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2472";>Spark-compatible
cast from integral to decimal types</a></li>
+<li><a href="https://github.com/apache/datafusion-comet/pull/2490";>Support for
decimal type to boolean cast</a></li>
+<li><a href="https://github.com/apache/datafusion-comet/pull/2316";>More date
part expressions</a></li>
+</ul>
+<h3 id="performance-improvements">Performance Improvements<a
class="headerlink" href="#performance-improvements" title="Permanent
link">¶</a></h3>
+<ul>
+<li><a href="https://github.com/apache/datafusion-comet/pull/2417";>Improved
BroadcastExchangeExec conversion</a> for better broadcast join performance</li>
+<li><a href="https://github.com/apache/datafusion-comet/pull/2407";>Use of
DataFusion's native <code>count_udaf</code></a> instead of <code>SUM(IF(expr IS
NOT NULL, 1, 0))</code></li>
+<li><a href="https://github.com/apache/datafusion-comet/pull/2402";>New
configuration from shared conf</a> to reduce overhead</li>
+<li><a href="https://github.com/apache/datafusion-comet/pull/2579";>Buffered
index writes</a> to reduce system calls in shuffle operations</li>
+</ul>
+<h3 id="comet-0110-tpc-h-performance">Comet 0.11.0 TPC-H Performance<a
class="headerlink" href="#comet-0110-tpc-h-performance" title="Permanent
link">¶</a></h3>
+<p>Comet 0.11.0 continues to deliver significant performance improvements over
Spark. In our <a
href="https://github.com/apache/datafusion-comet/pull/2596";>TPC-H
benchmarks</a>, Comet reduced overall query runtime from 687 seconds to 302
seconds when processing 100 GB of Parquet data using a single 8-core executor,
achieving a <strong>2.2x speedup</strong>.</p>
+<p><img alt="TPC-H Overall Performance"
src="/images/comet-0.11.0/tpch_allqueries.png"/></p>
+<p>The performance gains are consistent across individual queries, with most
queries showing substantial improvements:</p>
+<p><img alt="TPC-H Query-by-Query Comparison"
src="/images/comet-0.11.0/tpch_queries_compare.png"/></p>
+<p>You can reproduce these benchmarks using our <a
href="https://datafusion.apache.org/comet/contributor-guide/benchmarking.html";>Comet
Benchmarking Guide</a>. We encourage you to run your own performance tests
with your workloads.</p>
+<h3 id="apache-iceberg-support">Apache Iceberg Support<a class="headerlink"
href="#apache-iceberg-support" title="Permanent link">¶</a></h3>
+<ul>
+<li><a href="https://github.com/apache/datafusion-comet/pull/2386";>Updated
support for Apache Iceberg 1.9.1</a></li>
+<li><a href="https://github.com/apache/datafusion-comet/pull/2442";>Additional
Parquet-independent API improvements</a> for Iceberg integration</li>
+<li><a href="https://github.com/apache/datafusion-comet/pull/2510";>Improved
resource management</a> in Iceberg reader instances</li>
+</ul>
+<h3 id="ux-improvements">UX Improvements<a class="headerlink"
href="#ux-improvements" title="Permanent link">¶</a></h3>
+<ul>
+<li><a href="https://github.com/apache/datafusion-comet/pull/2412";>Added plan
conversion statistics to extended explain info</a> for better observability</li>
+<li><a href="https://github.com/apache/datafusion-comet/pull/2450";>Improved
fallback information</a> to help users understand when and why Comet falls back
to Spark</li>
+<li><a href="https://github.com/apache/datafusion-comet/pull/2515";>Added
<code>backtrace</code> feature</a> to simplify enabling native backtraces in
<code>CometNativeException</code></li>
+<li><a href="https://github.com/apache/datafusion-comet/pull/2379";>Native log
level is now configurable</a> via Comet configuration</li>
+</ul>
+<h3 id="bug-fixes">Bug Fixes<a class="headerlink" href="#bug-fixes"
title="Permanent link">¶</a></h3>
+<ul>
+<li><a href="https://github.com/apache/datafusion-comet/pull/2398";>Resolved
issues with reused broadcast plans in non-AQE mode</a></li>
+<li><a href="https://github.com/apache/datafusion-comet/pull/2420";>Fixed
thread safety in setNumPartitions</a></li>
+<li><a href="https://github.com/apache/datafusion-comet/pull/2440";>Improved
error handling when resolving S3 bucket region</a></li>
+<li><a href="https://github.com/apache/datafusion-comet/pull/2432";>Fixed byte
array literal casting issues</a></li>
+<li><a href="https://github.com/apache/datafusion-comet/pull/2438";>Corrected
subquery filter pushdown behavior for native_datafusion scan</a></li>
+</ul>
+<h3 id="documentation-updates">Documentation Updates<a class="headerlink"
href="#documentation-updates" title="Permanent link">¶</a></h3>
+<ul>
+<li><a href="https://github.com/apache/datafusion-comet/pull/2487";>Updated
documentation for native shuffle configuration and tuning</a></li>
+<li><a href="https://github.com/apache/datafusion-comet/pull/2496";>Added
documentation for ANSI mode support</a> in various functions</li>
+<li><a href="https://github.com/apache/datafusion-comet/pull/2474";>Improved
EC2 benchmarking guide</a></li>
+<li><a href="https://github.com/apache/datafusion-comet/pull/2568";>Split
configuration guide into different sections</a> (scan, exec, shuffle, etc.) for
better organization</li>
+<li>Various clarifications and improvements throughout the documentation</li>
+</ul>
+<h3 id="spark-compatibility">Spark Compatibility<a class="headerlink"
href="#spark-compatibility" title="Permanent link">¶</a></h3>
+<ul>
+<li>Spark 3.4.3 with JDK 11 & 17, Scala 2.12 & 2.13</li>
+<li>Spark 3.5.4 through 3.5.6 with JDK 11 & 17, Scala 2.12 & 2.13</li>
+<li>Spark 4.0.1 with JDK 17, Scala 2.13</li>
+</ul>
+<p>We are looking for help from the community to fully support Spark 4.0.1.
See <a href="https://github.com/apache/datafusion-comet/issues/1637";>EPIC:
Support 4.0.0</a> for more information.</p>
+<h2 id="getting-involved">Getting Involved<a class="headerlink"
href="#getting-involved" title="Permanent link">¶</a></h2>
+<p>The Comet project welcomes new contributors. We use the same <a
href="https://datafusion.apache.org/contributor-guide/communication.html#slack-and-discord";>Slack
and Discord</a> channels as the main DataFusion
+project and have a weekly <a
href="https://docs.google.com/document/d/1NBpkIAuU7O9h8Br5CbFksDhX-L9TyO9wmGLPMe0Plc8/edit?usp=sharing";>DataFusion
video call</a>.</p>
+<p>The easiest way to get involved is to test Comet with your current Spark
jobs and file issues for any bugs or
+performance regressions that you find. See the <a
href="https://datafusion.apache.org/comet/user-guide/installation.html";>Getting
Started</a> guide for instructions on downloading and installing
+Comet.</p>
+<p>There are also many <a
href="https://github.com/apache/datafusion-comet/contribute";>good first
issues</a> waiting for contributions.</p>
+
+<!--
+ Comments Section
+ Loaded only after explicit visitor consent to comply with ASF policy.
+-->
+
+<div id="comments">
+ <hr>
+ <h3>Comments</h3>
+
+ <!-- Local loader script -->
+ <script src="/content/js/giscus-consent.js" defer></script>
+
+ <!-- Consent UI -->
+ <div id="giscus-consent">
+ <p>
+ We use <a href="https://giscus.app/";>Giscus</a> for comments, powered
by GitHub Discussions.
+ To respect your privacy, Giscus and comments will load only if you
click "Show Comments"
+ </p>
+
+ <div class="consent-actions">
+ <button id="giscus-load" type="button">Show Comments</button>
+ <button id="giscus-revoke" type="button" hidden>Hide Comments</button>
+ </div>
+
+ <noscript>JavaScript is required to load comments from Giscus.</noscript>
+ </div>
+
+ <!-- Container where Giscus will render -->
+ <div id="comment-thread"></div>
+</div> </div>
+ <aside class="toc-container d-none d-md-block col-md-4 col-xl-3 ms-xl-2">
+ <div class="toc"><span class="toctitle">Contents</span><ul>
+<li><a href="#release-highlights">Release Highlights</a><ul>
+<li><a href="#parquet-modular-encryption-support">Parquet Modular Encryption
Support</a></li>
+<li><a href="#improved-memory-management">Improved Memory Management</a></li>
+<li><a href="#improved-apache-spark-40-support">Improved Apache Spark 4.0
Support</a></li>
+<li><a href="#complex-types-for-columnar-shuffle">Complex Types for Columnar
Shuffle</a></li>
+<li><a href="#rangepartitioning-for-native-shuffle">RangePartitioning for
Native Shuffle</a></li>
+<li><a href="#new-functionality">New Functionality</a></li>
+<li><a href="#performance-improvements">Performance Improvements</a></li>
+<li><a href="#comet-0110-tpc-h-performance">Comet 0.11.0 TPC-H
Performance</a></li>
+<li><a href="#apache-iceberg-support">Apache Iceberg Support</a></li>
+<li><a href="#ux-improvements">UX Improvements</a></li>
+<li><a href="#bug-fixes">Bug Fixes</a></li>
+<li><a href="#documentation-updates">Documentation Updates</a></li>
+<li><a href="#spark-compatibility">Spark Compatibility</a></li>
+</ul>
+</li>
+<li><a href="#getting-involved">Getting Involved</a></li>
+</ul>
+</div>
+ </aside>
+ </div>
+ </div>
+</div>
+ <!-- footer -->
+ <div class="row g-0">
+ <div class="col-12">
+ <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
+ Copyright 2025, <a href="https://www.apache.org/";>The Apache
Software Foundation</a>, Licensed under the <a
href="https://www.apache.org/licenses/LICENSE-2.0";>Apache License, Version
2.0</a>.<br/>
+ Apache® and the Apache feather logo are trademarks of The Apache
Software Foundation.
+ </p>
+ </div>
+ </div>
+ <script src="/blog/js/bootstrap.bundle.min.js"></script> </main>
+ </body>
+</html>
diff --git a/output/author/pmc.html b/output/author/pmc.html
index 79ea081..d112de6 100644
--- a/output/author/pmc.html
+++ b/output/author/pmc.html
@@ -20,6 +20,38 @@
<h2>Articles by pmc</h2>
<ol id="post-list">
+ <li><article class="hentry">
+ <header> <h2 class="entry-title"><a
href="https://datafusion.apache.org/blog/2025/10/21/datafusion-comet-0.11.0";
rel="bookmark" title="Permalink to Apache DataFusion Comet 0.11.0
Release">Apache DataFusion Comet 0.11.0 Release</a></h2> </header>
+ <footer class="post-info">
+ <time class="published"
datetime="2025-10-21T00:00:00+00:00"> Tue 21 October 2025 </time>
+ <address class="vcard author">By
+ <a class="url fn"
href="https://datafusion.apache.org/blog/author/pmc.html";>pmc</a>
+ </address>
+ </footer><!-- /.post-info -->
+ <div class="entry-content"> <!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<p>The Apache DataFusion PMC is pleased to announce version 0.11.0 of the <a
href="https://datafusion.apache.org/comet/";>Comet</a> subproject.</p>
+<p>Comet is an accelerator for Apache Spark that translates Spark physical
plans to DataFusion physical plans for
+improved performance and efficiency without requiring any code changes.</p>
+<p>This release covers approximately five weeks of development …</p>
</div><!-- /.entry-content -->
+ </article></li>
<li><article class="hentry">
<header> <h2 class="entry-title"><a
href="https://datafusion.apache.org/blog/2025/09/29/datafusion-50.0.0";
rel="bookmark" title="Permalink to Apache DataFusion 50.0.0 Released">Apache
DataFusion 50.0.0 Released</a></h2> </header>
<footer class="post-info">
diff --git a/output/category/blog.html b/output/category/blog.html
index 45fc70a..7257a5a 100644
--- a/output/category/blog.html
+++ b/output/category/blog.html
@@ -21,6 +21,38 @@
<h2>Articles in the blog category</h2>
<ol id="post-list">
+ <li><article class="hentry">
+ <header> <h2 class="entry-title"><a
href="https://datafusion.apache.org/blog/2025/10/21/datafusion-comet-0.11.0";
rel="bookmark" title="Permalink to Apache DataFusion Comet 0.11.0
Release">Apache DataFusion Comet 0.11.0 Release</a></h2> </header>
+ <footer class="post-info">
+ <time class="published"
datetime="2025-10-21T00:00:00+00:00"> Tue 21 October 2025 </time>
+ <address class="vcard author">By
+ <a class="url fn"
href="https://datafusion.apache.org/blog/author/pmc.html";>pmc</a>
+ </address>
+ </footer><!-- /.post-info -->
+ <div class="entry-content"> <!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<p>The Apache DataFusion PMC is pleased to announce version 0.11.0 of the <a
href="https://datafusion.apache.org/comet/";>Comet</a> subproject.</p>
+<p>Comet is an accelerator for Apache Spark that translates Spark physical
plans to DataFusion physical plans for
+improved performance and efficiency without requiring any code changes.</p>
+<p>This release covers approximately five weeks of development …</p>
</div><!-- /.entry-content -->
+ </article></li>
<li><article class="hentry">
<header> <h2 class="entry-title"><a
href="https://datafusion.apache.org/blog/2025/09/29/datafusion-50.0.0";
rel="bookmark" title="Permalink to Apache DataFusion 50.0.0 Released">Apache
DataFusion 50.0.0 Released</a></h2> </header>
<footer class="post-info">
diff --git a/output/feed.xml b/output/feed.xml
index 43b2d08..2579a43 100644
--- a/output/feed.xml
+++ b/output/feed.xml
@@ -1,5 +1,27 @@
<?xml version="1.0" encoding="utf-8"?>
-<rss version="2.0"><channel><title>Apache DataFusion
Blog</title><link>https://datafusion.apache.org/blog/</link><description></description><lastBuildDate>Mon,
29 Sep 2025 00:00:00 +0000</lastBuildDate><item><title>Apache DataFusion
50.0.0
Released</title><link>https://datafusion.apache.org/blog/2025/09/29/datafusion-50.0.0</link><description><!--
+<rss version="2.0"><channel><title>Apache DataFusion
Blog</title><link>https://datafusion.apache.org/blog/</link><description></description><lastBuildDate>Tue,
21 Oct 2025 00:00:00 +0000</lastBuildDate><item><title>Apache DataFusion Comet
0.11.0
Release</title><link>https://datafusion.apache.org/blog/2025/10/21/datafusion-comet-0.11.0</link><description><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<p>The Apache DataFusion PMC is pleased to announce version 0.11.0 of
the <a href="https://datafusion.apache.org/comet/">Comet</a>
subproject.</p>
+<p>Comet is an accelerator for Apache Spark that translates Spark
physical plans to DataFusion physical plans for
+improved performance and efficiency without requiring any code
changes.</p>
+<p>This release covers approximately five weeks of development
…</p></description><dc:creator
xmlns:dc="http://purl.org/dc/elements/1.1/";>pmc</dc:creator><pubDate>Tue, 21
Oct 2025 00:00:00 +0000</pubDate><guid
isPermaLink="false">tag:datafusion.apache.org,2025-10-21:/blog/2025/10/21/datafusion-comet-0.11.0</guid><category>blog</category></item><item><title>Apache
DataFusion 50.0.0
Released</title><link>https://datafusion.apache.org/blog/2025/09/29/datafusion-50.0.0</link><des
[...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
diff --git a/output/feeds/all-en.atom.xml b/output/feeds/all-en.atom.xml
index 5e381b8..ab443c1 100644
--- a/output/feeds/all-en.atom.xml
+++ b/output/feeds/all-en.atom.xml
@@ -1,5 +1,152 @@
<?xml version="1.0" encoding="utf-8"?>
-<feed xmlns="http://www.w3.org/2005/Atom";><title>Apache DataFusion
Blog</title><link href="https://datafusion.apache.org/blog/";
rel="alternate"></link><link
href="https://datafusion.apache.org/blog/feeds/all-en.atom.xml";
rel="self"></link><id>https://datafusion.apache.org/blog/</id><updated>2025-09-29T00:00:00+00:00</updated><subtitle></subtitle><entry><title>Apache
DataFusion 50.0.0 Released</title><link
href="https://datafusion.apache.org/blog/2025/09/29/datafusion-50.0.0";
rel="alterna [...]
+<feed xmlns="http://www.w3.org/2005/Atom";><title>Apache DataFusion
Blog</title><link href="https://datafusion.apache.org/blog/";
rel="alternate"></link><link
href="https://datafusion.apache.org/blog/feeds/all-en.atom.xml";
rel="self"></link><id>https://datafusion.apache.org/blog/</id><updated>2025-10-21T00:00:00+00:00</updated><subtitle></subtitle><entry><title>Apache
DataFusion Comet 0.11.0 Release</title><link
href="https://datafusion.apache.org/blog/2025/10/21/datafusion-comet-0.11.0"; r
[...]
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<p>The Apache DataFusion PMC is pleased to announce version 0.11.0 of
the <a href="https://datafusion.apache.org/comet/">Comet</a>
subproject.</p>
+<p>Comet is an accelerator for Apache Spark that translates Spark
physical plans to DataFusion physical plans for
+improved performance and efficiency without requiring any code
changes.</p>
+<p>This release covers approximately five weeks of development
…</p></summary><content type="html"><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<p>The Apache DataFusion PMC is pleased to announce version 0.11.0 of
the <a href="https://datafusion.apache.org/comet/">Comet</a>
subproject.</p>
+<p>Comet is an accelerator for Apache Spark that translates Spark
physical plans to DataFusion physical plans for
+improved performance and efficiency without requiring any code
changes.</p>
+<p>This release covers approximately five weeks of development work and
is the result of merging 131 PRs from 15
+contributors. See the <a
href="https://github.com/apache/datafusion-comet/blob/main/dev/changelog/0.11.0.md">change
log</a> for more information.</p>
+<h2 id="release-highlights">Release Highlights<a class="headerlink"
href="#release-highlights" title="Permanent link">¶</a></h2>
+<h3 id="parquet-modular-encryption-support">Parquet Modular Encryption
Support<a class="headerlink" href="#parquet-modular-encryption-support"
title="Permanent link">¶</a></h3>
+<p>Spark supports Parquet Modular Encryption to independently encrypt
column values and metadata. Furthermore, Spark supports custom encryption
factories for users to provide their own key-management service (KMS)
implementations. Thanks to <a
href="https://github.com/apache/arrow-rs/issues/7278">a</a> <a
href="https://github.com/apache/datafusion/issues/15216">number</a>
<a href="https://github.com/apache/datafusion/pull/16351">of</a>
<a href [...]
+<h3 id="improved-memory-management">Improved Memory Management<a
class="headerlink" href="#improved-memory-management" title="Permanent
link">¶</a></h3>
+<p>Comet 0.11.0 introduces significant improvements to memory
management, making it easier to deploy and more resilient to out-of-memory
conditions:</p>
+<ul>
+<li><strong>Changed default memory pool</strong>: The
default off-heap memory pool has been <a
href="https://github.com/apache/datafusion-comet/pull/2526">changed from
<code>greedy_unified</code> to
<code>fair_unified</code></a>, providing better memory
fairness across operations</li>
+<li><strong>Off-heap deployment recommended</strong>: To
simplify configuration and improve performance, Comet now expects to be
deployed with Spark's off-heap memory configuration. <a
href="https://github.com/apache/datafusion-comet/pull/2554">On-heap memory
is still available</a> for development and debugging, but is not
recommended for deployment</li>
+<li><strong>Better disk management</strong>: The <a
href="https://github.com/apache/datafusion-comet/pull/2479">DiskManager
<code>max_temp_directory_size</code> is now configurable</a>
for better control over temporary disk usage</li>
+<li><strong>Enhanced safety</strong>: Memory pool operations
now <a href="https://github.com/apache/datafusion-comet/pull/2455">use
checked arithmetic operations</a> to prevent overflow issues</li>
+</ul>
+<p>These changes make Comet significantly easier to configure and deploy
in production environments.</p>
+<h3 id="improved-apache-spark-40-support">Improved Apache Spark 4.0
Support<a class="headerlink" href="#improved-apache-spark-40-support"
title="Permanent link">¶</a></h3>
+<p>Comet has improved its support for Apache Spark 4.0.1 with several
important enhancements:</p>
+<ul>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2414">Updated support
from Spark 4.0.0 to Spark 4.0.1</a></li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2514">Spark 4.0 is now
included in the release build script</a></li>
+<li>Expanded ANSI mode compatibility with several new
implementations:</li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2136">ANSI evaluation
mode arithmetic operations</a></li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2421">ANSI mode
integral divide</a></li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2542">ANSI mode
rounding functions</a></li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2556">ANSI mode
remainder function</a></li>
+</ul>
+<p>Spark 4.0 compatible jar files are now available on Maven Central.
See the <a
href="https://datafusion.apache.org/comet/user-guide/0.11/installation.html#using-a-published-jar-file">installation
guide</a> for instructions on using published jar files.</p>
+<h3 id="complex-types-for-columnar-shuffle">Complex Types for Columnar
Shuffle<a class="headerlink" href="#complex-types-for-columnar-shuffle"
title="Permanent link">¶</a></h3>
+<p><a href="https://github.com/ashdnazg">ashdnazg</a>
submitted a <a
href="https://github.com/apache/datafusion-comet/pull/2571">fantastic
refactoring PR</a> that simplified the logic for writing rows in Comet’s
JVM-based, columnar shuffle. A benefit of this refactoring is better support
for complex types (<em>e.g.,</em> structs, lists, and arrays) in
columnar shuffle. Comet no longer falls back to Spark to shuffle these types,
enabling native ac [...]
+<h3 id="rangepartitioning-for-native-shuffle">RangePartitioning for
Native Shuffle<a class="headerlink"
href="#rangepartitioning-for-native-shuffle" title="Permanent
link">¶</a></h3>
+<p>Comet's native shuffle now <a
href="https://github.com/apache/datafusion-comet/pull/2258">supports
RangePartitioning</a>, providing better performance for operations that
require range-based data distribution. Comet now matches Spark behavior for
computing and distributing range boundaries, and serializes them to native
execution for faster shuffle operations.</p>
+<h3 id="new-functionality">New Functionality<a class="headerlink"
href="#new-functionality" title="Permanent link">¶</a></h3>
+<p>The following SQL functions are now supported:</p>
+<ul>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2411"><code>weekday</code></a>
- Extract day of week from date</li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2102"><code>lpad</code></a>
- Left pad a string with column support for pad length</li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2099"><code>rpad</code></a>
- Right pad a string with <a
href="https://github.com/apache/datafusion-comet/pull/2436">column support
and additional character support</a></li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2481"><code>reverse</code></a>
- Support for ArrayType input in addition to strings</li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2429"><code>count(distinct)</code></a>
- Native support without falling back to Spark</li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2466"><code>bit_get</code></a>
- Get bit value at position</li>
+</ul>
+<p>New expression capabilities include:</p>
+<ul>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2181">Nested array
literal support</a></li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2425">Array-to-string
cast support</a></li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2472">Spark-compatible
cast from integral to decimal types</a></li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2490">Support for
decimal type to boolean cast</a></li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2316">More date part
expressions</a></li>
+</ul>
+<h3 id="performance-improvements">Performance Improvements<a
class="headerlink" href="#performance-improvements" title="Permanent
link">¶</a></h3>
+<ul>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2417">Improved
BroadcastExchangeExec conversion</a> for better broadcast join
performance</li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2407">Use of
DataFusion's native <code>count_udaf</code></a> instead of
<code>SUM(IF(expr IS NOT NULL, 1, 0))</code></li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2402">New
configuration from shared conf</a> to reduce overhead</li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2579">Buffered index
writes</a> to reduce system calls in shuffle operations</li>
+</ul>
+<h3 id="comet-0110-tpc-h-performance">Comet 0.11.0 TPC-H
Performance<a class="headerlink" href="#comet-0110-tpc-h-performance"
title="Permanent link">¶</a></h3>
+<p>Comet 0.11.0 continues to deliver significant performance
improvements over Spark. In our <a
href="https://github.com/apache/datafusion-comet/pull/2596">TPC-H
benchmarks</a>, Comet reduced overall query runtime from 687 seconds to
302 seconds when processing 100 GB of Parquet data using a single 8-core
executor, achieving a <strong>2.2x speedup</strong>.</p>
+<p><img alt="TPC-H Overall Performance"
src="/images/comet-0.11.0/tpch_allqueries.png"/></p>
+<p>The performance gains are consistent across individual queries, with
most queries showing substantial improvements:</p>
+<p><img alt="TPC-H Query-by-Query Comparison"
src="/images/comet-0.11.0/tpch_queries_compare.png"/></p>
+<p>You can reproduce these benchmarks using our <a
href="https://datafusion.apache.org/comet/contributor-guide/benchmarking.html">Comet
Benchmarking Guide</a>. We encourage you to run your own performance
tests with your workloads.</p>
+<h3 id="apache-iceberg-support">Apache Iceberg Support<a
class="headerlink" href="#apache-iceberg-support" title="Permanent
link">¶</a></h3>
+<ul>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2386">Updated support
for Apache Iceberg 1.9.1</a></li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2442">Additional
Parquet-independent API improvements</a> for Iceberg
integration</li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2510">Improved
resource management</a> in Iceberg reader instances</li>
+</ul>
+<h3 id="ux-improvements">UX Improvements<a class="headerlink"
href="#ux-improvements" title="Permanent link">¶</a></h3>
+<ul>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2412">Added plan
conversion statistics to extended explain info</a> for better
observability</li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2450">Improved
fallback information</a> to help users understand when and why Comet
falls back to Spark</li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2515">Added
<code>backtrace</code> feature</a> to simplify enabling
native backtraces in <code>CometNativeException</code></li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2379">Native log level
is now configurable</a> via Comet configuration</li>
+</ul>
+<h3 id="bug-fixes">Bug Fixes<a class="headerlink" href="#bug-fixes"
title="Permanent link">¶</a></h3>
+<ul>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2398">Resolved issues
with reused broadcast plans in non-AQE mode</a></li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2420">Fixed thread
safety in setNumPartitions</a></li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2440">Improved error
handling when resolving S3 bucket region</a></li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2432">Fixed byte array
literal casting issues</a></li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2438">Corrected
subquery filter pushdown behavior for native_datafusion
scan</a></li>
+</ul>
+<h3 id="documentation-updates">Documentation Updates<a
class="headerlink" href="#documentation-updates" title="Permanent
link">¶</a></h3>
+<ul>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2487">Updated
documentation for native shuffle configuration and tuning</a></li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2496">Added
documentation for ANSI mode support</a> in various functions</li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2474">Improved EC2
benchmarking guide</a></li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2568">Split
configuration guide into different sections</a> (scan, exec, shuffle,
etc.) for better organization</li>
+<li>Various clarifications and improvements throughout the
documentation</li>
+</ul>
+<h3 id="spark-compatibility">Spark Compatibility<a class="headerlink"
href="#spark-compatibility" title="Permanent link">¶</a></h3>
+<ul>
+<li>Spark 3.4.3 with JDK 11 &amp; 17, Scala 2.12 &amp;
2.13</li>
+<li>Spark 3.5.4 through 3.5.6 with JDK 11 &amp; 17, Scala 2.12
&amp; 2.13</li>
+<li>Spark 4.0.1 with JDK 17, Scala 2.13</li>
+</ul>
+<p>We are looking for help from the community to fully support Spark
4.0.1. See <a
href="https://github.com/apache/datafusion-comet/issues/1637">EPIC: Support
4.0.0</a> for more information.</p>
+<h2 id="getting-involved">Getting Involved<a class="headerlink"
href="#getting-involved" title="Permanent link">¶</a></h2>
+<p>The Comet project welcomes new contributors. We use the same <a
href="https://datafusion.apache.org/contributor-guide/communication.html#slack-and-discord">Slack
and Discord</a> channels as the main DataFusion
+project and have a weekly <a
href="https://docs.google.com/document/d/1NBpkIAuU7O9h8Br5CbFksDhX-L9TyO9wmGLPMe0Plc8/edit?usp=sharing">DataFusion
video call</a>.</p>
+<p>The easiest way to get involved is to test Comet with your current
Spark jobs and file issues for any bugs or
+performance regressions that you find. See the <a
href="https://datafusion.apache.org/comet/user-guide/installation.html">Getting
Started</a> guide for instructions on downloading and installing
+Comet.</p>
+<p>There are also many <a
href="https://github.com/apache/datafusion-comet/contribute">good first
issues</a> waiting for contributions.</p></content><category
term="blog"></category></entry><entry><title>Apache DataFusion 50.0.0
Released</title><link
href="https://datafusion.apache.org/blog/2025/09/29/datafusion-50.0.0";
rel="alternate"></link><published>2025-09-29T00:00:00+00:00</published><updated>2025-09-29T00:00:00+00:00</updated><author><name>pmc</name></autho
[...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
diff --git a/output/feeds/blog.atom.xml b/output/feeds/blog.atom.xml
index e54b3e1..89c1804 100644
--- a/output/feeds/blog.atom.xml
+++ b/output/feeds/blog.atom.xml
@@ -1,5 +1,152 @@
<?xml version="1.0" encoding="utf-8"?>
-<feed xmlns="http://www.w3.org/2005/Atom";><title>Apache DataFusion Blog -
blog</title><link href="https://datafusion.apache.org/blog/";
rel="alternate"></link><link
href="https://datafusion.apache.org/blog/feeds/blog.atom.xml";
rel="self"></link><id>https://datafusion.apache.org/blog/</id><updated>2025-09-29T00:00:00+00:00</updated><subtitle></subtitle><entry><title>Apache
DataFusion 50.0.0 Released</title><link
href="https://datafusion.apache.org/blog/2025/09/29/datafusion-50.0.0"; rel="al
[...]
+<feed xmlns="http://www.w3.org/2005/Atom";><title>Apache DataFusion Blog -
blog</title><link href="https://datafusion.apache.org/blog/";
rel="alternate"></link><link
href="https://datafusion.apache.org/blog/feeds/blog.atom.xml";
rel="self"></link><id>https://datafusion.apache.org/blog/</id><updated>2025-10-21T00:00:00+00:00</updated><subtitle></subtitle><entry><title>Apache
DataFusion Comet 0.11.0 Release</title><link
href="https://datafusion.apache.org/blog/2025/10/21/datafusion-comet-0.11 [...]
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<p>The Apache DataFusion PMC is pleased to announce version 0.11.0 of
the <a href="https://datafusion.apache.org/comet/">Comet</a>
subproject.</p>
+<p>Comet is an accelerator for Apache Spark that translates Spark
physical plans to DataFusion physical plans for
+improved performance and efficiency without requiring any code
changes.</p>
+<p>This release covers approximately five weeks of development
…</p></summary><content type="html"><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<p>The Apache DataFusion PMC is pleased to announce version 0.11.0 of
the <a href="https://datafusion.apache.org/comet/">Comet</a>
subproject.</p>
+<p>Comet is an accelerator for Apache Spark that translates Spark
physical plans to DataFusion physical plans for
+improved performance and efficiency without requiring any code
changes.</p>
+<p>This release covers approximately five weeks of development work and
is the result of merging 131 PRs from 15
+contributors. See the <a
href="https://github.com/apache/datafusion-comet/blob/main/dev/changelog/0.11.0.md">change
log</a> for more information.</p>
+<h2 id="release-highlights">Release Highlights<a class="headerlink"
href="#release-highlights" title="Permanent link">¶</a></h2>
+<h3 id="parquet-modular-encryption-support">Parquet Modular Encryption
Support<a class="headerlink" href="#parquet-modular-encryption-support"
title="Permanent link">¶</a></h3>
+<p>Spark supports Parquet Modular Encryption to independently encrypt
column values and metadata. Furthermore, Spark supports custom encryption
factories for users to provide their own key-management service (KMS)
implementations. Thanks to <a
href="https://github.com/apache/arrow-rs/issues/7278">a</a> <a
href="https://github.com/apache/datafusion/issues/15216">number</a>
<a href="https://github.com/apache/datafusion/pull/16351">of</a>
<a href [...]
+<h3 id="improved-memory-management">Improved Memory Management<a
class="headerlink" href="#improved-memory-management" title="Permanent
link">¶</a></h3>
+<p>Comet 0.11.0 introduces significant improvements to memory
management, making it easier to deploy and more resilient to out-of-memory
conditions:</p>
+<ul>
+<li><strong>Changed default memory pool</strong>: The
default off-heap memory pool has been <a
href="https://github.com/apache/datafusion-comet/pull/2526">changed from
<code>greedy_unified</code> to
<code>fair_unified</code></a>, providing better memory
fairness across operations</li>
+<li><strong>Off-heap deployment recommended</strong>: To
simplify configuration and improve performance, Comet now expects to be
deployed with Spark's off-heap memory configuration. <a
href="https://github.com/apache/datafusion-comet/pull/2554">On-heap memory
is still available</a> for development and debugging, but is not
recommended for deployment</li>
+<li><strong>Better disk management</strong>: The <a
href="https://github.com/apache/datafusion-comet/pull/2479">DiskManager
<code>max_temp_directory_size</code> is now configurable</a>
for better control over temporary disk usage</li>
+<li><strong>Enhanced safety</strong>: Memory pool operations
now <a href="https://github.com/apache/datafusion-comet/pull/2455">use
checked arithmetic operations</a> to prevent overflow issues</li>
+</ul>
+<p>These changes make Comet significantly easier to configure and deploy
in production environments.</p>
+<h3 id="improved-apache-spark-40-support">Improved Apache Spark 4.0
Support<a class="headerlink" href="#improved-apache-spark-40-support"
title="Permanent link">¶</a></h3>
+<p>Comet has improved its support for Apache Spark 4.0.1 with several
important enhancements:</p>
+<ul>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2414">Updated support
from Spark 4.0.0 to Spark 4.0.1</a></li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2514">Spark 4.0 is now
included in the release build script</a></li>
+<li>Expanded ANSI mode compatibility with several new
implementations:</li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2136">ANSI evaluation
mode arithmetic operations</a></li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2421">ANSI mode
integral divide</a></li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2542">ANSI mode
rounding functions</a></li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2556">ANSI mode
remainder function</a></li>
+</ul>
+<p>Spark 4.0 compatible jar files are now available on Maven Central.
See the <a
href="https://datafusion.apache.org/comet/user-guide/0.11/installation.html#using-a-published-jar-file">installation
guide</a> for instructions on using published jar files.</p>
+<h3 id="complex-types-for-columnar-shuffle">Complex Types for Columnar
Shuffle<a class="headerlink" href="#complex-types-for-columnar-shuffle"
title="Permanent link">¶</a></h3>
+<p><a href="https://github.com/ashdnazg">ashdnazg</a>
submitted a <a
href="https://github.com/apache/datafusion-comet/pull/2571">fantastic
refactoring PR</a> that simplified the logic for writing rows in Comet’s
JVM-based, columnar shuffle. A benefit of this refactoring is better support
for complex types (<em>e.g.,</em> structs, lists, and arrays) in
columnar shuffle. Comet no longer falls back to Spark to shuffle these types,
enabling native ac [...]
+<h3 id="rangepartitioning-for-native-shuffle">RangePartitioning for
Native Shuffle<a class="headerlink"
href="#rangepartitioning-for-native-shuffle" title="Permanent
link">¶</a></h3>
+<p>Comet's native shuffle now <a
href="https://github.com/apache/datafusion-comet/pull/2258">supports
RangePartitioning</a>, providing better performance for operations that
require range-based data distribution. Comet now matches Spark behavior for
computing and distributing range boundaries, and serializes them to native
execution for faster shuffle operations.</p>
+<h3 id="new-functionality">New Functionality<a class="headerlink"
href="#new-functionality" title="Permanent link">¶</a></h3>
+<p>The following SQL functions are now supported:</p>
+<ul>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2411"><code>weekday</code></a>
- Extract day of week from date</li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2102"><code>lpad</code></a>
- Left pad a string with column support for pad length</li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2099"><code>rpad</code></a>
- Right pad a string with <a
href="https://github.com/apache/datafusion-comet/pull/2436">column support
and additional character support</a></li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2481"><code>reverse</code></a>
- Support for ArrayType input in addition to strings</li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2429"><code>count(distinct)</code></a>
- Native support without falling back to Spark</li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2466"><code>bit_get</code></a>
- Get bit value at position</li>
+</ul>
+<p>New expression capabilities include:</p>
+<ul>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2181">Nested array
literal support</a></li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2425">Array-to-string
cast support</a></li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2472">Spark-compatible
cast from integral to decimal types</a></li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2490">Support for
decimal type to boolean cast</a></li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2316">More date part
expressions</a></li>
+</ul>
+<h3 id="performance-improvements">Performance Improvements<a
class="headerlink" href="#performance-improvements" title="Permanent
link">¶</a></h3>
+<ul>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2417">Improved
BroadcastExchangeExec conversion</a> for better broadcast join
performance</li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2407">Use of
DataFusion's native <code>count_udaf</code></a> instead of
<code>SUM(IF(expr IS NOT NULL, 1, 0))</code></li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2402">New
configuration from shared conf</a> to reduce overhead</li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2579">Buffered index
writes</a> to reduce system calls in shuffle operations</li>
+</ul>
+<h3 id="comet-0110-tpc-h-performance">Comet 0.11.0 TPC-H
Performance<a class="headerlink" href="#comet-0110-tpc-h-performance"
title="Permanent link">¶</a></h3>
+<p>Comet 0.11.0 continues to deliver significant performance
improvements over Spark. In our <a
href="https://github.com/apache/datafusion-comet/pull/2596">TPC-H
benchmarks</a>, Comet reduced overall query runtime from 687 seconds to
302 seconds when processing 100 GB of Parquet data using a single 8-core
executor, achieving a <strong>2.2x speedup</strong>.</p>
+<p><img alt="TPC-H Overall Performance"
src="/images/comet-0.11.0/tpch_allqueries.png"/></p>
+<p>The performance gains are consistent across individual queries, with
most queries showing substantial improvements:</p>
+<p><img alt="TPC-H Query-by-Query Comparison"
src="/images/comet-0.11.0/tpch_queries_compare.png"/></p>
+<p>You can reproduce these benchmarks using our <a
href="https://datafusion.apache.org/comet/contributor-guide/benchmarking.html">Comet
Benchmarking Guide</a>. We encourage you to run your own performance
tests with your workloads.</p>
+<h3 id="apache-iceberg-support">Apache Iceberg Support<a
class="headerlink" href="#apache-iceberg-support" title="Permanent
link">¶</a></h3>
+<ul>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2386">Updated support
for Apache Iceberg 1.9.1</a></li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2442">Additional
Parquet-independent API improvements</a> for Iceberg
integration</li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2510">Improved
resource management</a> in Iceberg reader instances</li>
+</ul>
+<h3 id="ux-improvements">UX Improvements<a class="headerlink"
href="#ux-improvements" title="Permanent link">¶</a></h3>
+<ul>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2412">Added plan
conversion statistics to extended explain info</a> for better
observability</li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2450">Improved
fallback information</a> to help users understand when and why Comet
falls back to Spark</li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2515">Added
<code>backtrace</code> feature</a> to simplify enabling
native backtraces in <code>CometNativeException</code></li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2379">Native log level
is now configurable</a> via Comet configuration</li>
+</ul>
+<h3 id="bug-fixes">Bug Fixes<a class="headerlink" href="#bug-fixes"
title="Permanent link">¶</a></h3>
+<ul>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2398">Resolved issues
with reused broadcast plans in non-AQE mode</a></li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2420">Fixed thread
safety in setNumPartitions</a></li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2440">Improved error
handling when resolving S3 bucket region</a></li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2432">Fixed byte array
literal casting issues</a></li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2438">Corrected
subquery filter pushdown behavior for native_datafusion
scan</a></li>
+</ul>
+<h3 id="documentation-updates">Documentation Updates<a
class="headerlink" href="#documentation-updates" title="Permanent
link">¶</a></h3>
+<ul>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2487">Updated
documentation for native shuffle configuration and tuning</a></li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2496">Added
documentation for ANSI mode support</a> in various functions</li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2474">Improved EC2
benchmarking guide</a></li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2568">Split
configuration guide into different sections</a> (scan, exec, shuffle,
etc.) for better organization</li>
+<li>Various clarifications and improvements throughout the
documentation</li>
+</ul>
+<h3 id="spark-compatibility">Spark Compatibility<a class="headerlink"
href="#spark-compatibility" title="Permanent link">¶</a></h3>
+<ul>
+<li>Spark 3.4.3 with JDK 11 &amp; 17, Scala 2.12 &amp;
2.13</li>
+<li>Spark 3.5.4 through 3.5.6 with JDK 11 &amp; 17, Scala 2.12
&amp; 2.13</li>
+<li>Spark 4.0.1 with JDK 17, Scala 2.13</li>
+</ul>
+<p>We are looking for help from the community to fully support Spark
4.0.1. See <a
href="https://github.com/apache/datafusion-comet/issues/1637">EPIC: Support
4.0.0</a> for more information.</p>
+<h2 id="getting-involved">Getting Involved<a class="headerlink"
href="#getting-involved" title="Permanent link">¶</a></h2>
+<p>The Comet project welcomes new contributors. We use the same <a
href="https://datafusion.apache.org/contributor-guide/communication.html#slack-and-discord">Slack
and Discord</a> channels as the main DataFusion
+project and have a weekly <a
href="https://docs.google.com/document/d/1NBpkIAuU7O9h8Br5CbFksDhX-L9TyO9wmGLPMe0Plc8/edit?usp=sharing">DataFusion
video call</a>.</p>
+<p>The easiest way to get involved is to test Comet with your current
Spark jobs and file issues for any bugs or
+performance regressions that you find. See the <a
href="https://datafusion.apache.org/comet/user-guide/installation.html">Getting
Started</a> guide for instructions on downloading and installing
+Comet.</p>
+<p>There are also many <a
href="https://github.com/apache/datafusion-comet/contribute">good first
issues</a> waiting for contributions.</p></content><category
term="blog"></category></entry><entry><title>Apache DataFusion 50.0.0
Released</title><link
href="https://datafusion.apache.org/blog/2025/09/29/datafusion-50.0.0";
rel="alternate"></link><published>2025-09-29T00:00:00+00:00</published><updated>2025-09-29T00:00:00+00:00</updated><author><name>pmc</name></autho
[...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
diff --git a/output/feeds/pmc.atom.xml b/output/feeds/pmc.atom.xml
index b16eb0f..01a644c 100644
--- a/output/feeds/pmc.atom.xml
+++ b/output/feeds/pmc.atom.xml
@@ -1,5 +1,152 @@
<?xml version="1.0" encoding="utf-8"?>
-<feed xmlns="http://www.w3.org/2005/Atom";><title>Apache DataFusion Blog -
pmc</title><link href="https://datafusion.apache.org/blog/";
rel="alternate"></link><link
href="https://datafusion.apache.org/blog/feeds/pmc.atom.xml";
rel="self"></link><id>https://datafusion.apache.org/blog/</id><updated>2025-09-29T00:00:00+00:00</updated><subtitle></subtitle><entry><title>Apache
DataFusion 50.0.0 Released</title><link
href="https://datafusion.apache.org/blog/2025/09/29/datafusion-50.0.0";
rel="alte [...]
+<feed xmlns="http://www.w3.org/2005/Atom";><title>Apache DataFusion Blog -
pmc</title><link href="https://datafusion.apache.org/blog/";
rel="alternate"></link><link
href="https://datafusion.apache.org/blog/feeds/pmc.atom.xml";
rel="self"></link><id>https://datafusion.apache.org/blog/</id><updated>2025-10-21T00:00:00+00:00</updated><subtitle></subtitle><entry><title>Apache
DataFusion Comet 0.11.0 Release</title><link
href="https://datafusion.apache.org/blog/2025/10/21/datafusion-comet-0.11.0
[...]
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<p>The Apache DataFusion PMC is pleased to announce version 0.11.0 of
the <a href="https://datafusion.apache.org/comet/">Comet</a>
subproject.</p>
+<p>Comet is an accelerator for Apache Spark that translates Spark
physical plans to DataFusion physical plans for
+improved performance and efficiency without requiring any code
changes.</p>
+<p>This release covers approximately five weeks of development
…</p></summary><content type="html"><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<p>The Apache DataFusion PMC is pleased to announce version 0.11.0 of
the <a href="https://datafusion.apache.org/comet/">Comet</a>
subproject.</p>
+<p>Comet is an accelerator for Apache Spark that translates Spark
physical plans to DataFusion physical plans for
+improved performance and efficiency without requiring any code
changes.</p>
+<p>This release covers approximately five weeks of development work and
is the result of merging 131 PRs from 15
+contributors. See the <a
href="https://github.com/apache/datafusion-comet/blob/main/dev/changelog/0.11.0.md">change
log</a> for more information.</p>
+<h2 id="release-highlights">Release Highlights<a class="headerlink"
href="#release-highlights" title="Permanent link">¶</a></h2>
+<h3 id="parquet-modular-encryption-support">Parquet Modular Encryption
Support<a class="headerlink" href="#parquet-modular-encryption-support"
title="Permanent link">¶</a></h3>
+<p>Spark supports Parquet Modular Encryption to independently encrypt
column values and metadata. Furthermore, Spark supports custom encryption
factories for users to provide their own key-management service (KMS)
implementations. Thanks to <a
href="https://github.com/apache/arrow-rs/issues/7278">a</a> <a
href="https://github.com/apache/datafusion/issues/15216">number</a>
<a href="https://github.com/apache/datafusion/pull/16351">of</a>
<a href [...]
+<h3 id="improved-memory-management">Improved Memory Management<a
class="headerlink" href="#improved-memory-management" title="Permanent
link">¶</a></h3>
+<p>Comet 0.11.0 introduces significant improvements to memory
management, making it easier to deploy and more resilient to out-of-memory
conditions:</p>
+<ul>
+<li><strong>Changed default memory pool</strong>: The
default off-heap memory pool has been <a
href="https://github.com/apache/datafusion-comet/pull/2526">changed from
<code>greedy_unified</code> to
<code>fair_unified</code></a>, providing better memory
fairness across operations</li>
+<li><strong>Off-heap deployment recommended</strong>: To
simplify configuration and improve performance, Comet now expects to be
deployed with Spark's off-heap memory configuration. <a
href="https://github.com/apache/datafusion-comet/pull/2554">On-heap memory
is still available</a> for development and debugging, but is not
recommended for deployment</li>
+<li><strong>Better disk management</strong>: The <a
href="https://github.com/apache/datafusion-comet/pull/2479">DiskManager
<code>max_temp_directory_size</code> is now configurable</a>
for better control over temporary disk usage</li>
+<li><strong>Enhanced safety</strong>: Memory pool operations
now <a href="https://github.com/apache/datafusion-comet/pull/2455">use
checked arithmetic operations</a> to prevent overflow issues</li>
+</ul>
+<p>These changes make Comet significantly easier to configure and deploy
in production environments.</p>
+<h3 id="improved-apache-spark-40-support">Improved Apache Spark 4.0
Support<a class="headerlink" href="#improved-apache-spark-40-support"
title="Permanent link">¶</a></h3>
+<p>Comet has improved its support for Apache Spark 4.0.1 with several
important enhancements:</p>
+<ul>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2414">Updated support
from Spark 4.0.0 to Spark 4.0.1</a></li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2514">Spark 4.0 is now
included in the release build script</a></li>
+<li>Expanded ANSI mode compatibility with several new
implementations:</li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2136">ANSI evaluation
mode arithmetic operations</a></li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2421">ANSI mode
integral divide</a></li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2542">ANSI mode
rounding functions</a></li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2556">ANSI mode
remainder function</a></li>
+</ul>
+<p>Spark 4.0 compatible jar files are now available on Maven Central.
See the <a
href="https://datafusion.apache.org/comet/user-guide/0.11/installation.html#using-a-published-jar-file">installation
guide</a> for instructions on using published jar files.</p>
+<h3 id="complex-types-for-columnar-shuffle">Complex Types for Columnar
Shuffle<a class="headerlink" href="#complex-types-for-columnar-shuffle"
title="Permanent link">¶</a></h3>
+<p><a href="https://github.com/ashdnazg">ashdnazg</a>
submitted a <a
href="https://github.com/apache/datafusion-comet/pull/2571">fantastic
refactoring PR</a> that simplified the logic for writing rows in Comet’s
JVM-based, columnar shuffle. A benefit of this refactoring is better support
for complex types (<em>e.g.,</em> structs, lists, and arrays) in
columnar shuffle. Comet no longer falls back to Spark to shuffle these types,
enabling native ac [...]
+<h3 id="rangepartitioning-for-native-shuffle">RangePartitioning for
Native Shuffle<a class="headerlink"
href="#rangepartitioning-for-native-shuffle" title="Permanent
link">¶</a></h3>
+<p>Comet's native shuffle now <a
href="https://github.com/apache/datafusion-comet/pull/2258">supports
RangePartitioning</a>, providing better performance for operations that
require range-based data distribution. Comet now matches Spark behavior for
computing and distributing range boundaries, and serializes them to native
execution for faster shuffle operations.</p>
+<h3 id="new-functionality">New Functionality<a class="headerlink"
href="#new-functionality" title="Permanent link">¶</a></h3>
+<p>The following SQL functions are now supported:</p>
+<ul>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2411"><code>weekday</code></a>
- Extract day of week from date</li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2102"><code>lpad</code></a>
- Left pad a string with column support for pad length</li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2099"><code>rpad</code></a>
- Right pad a string with <a
href="https://github.com/apache/datafusion-comet/pull/2436">column support
and additional character support</a></li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2481"><code>reverse</code></a>
- Support for ArrayType input in addition to strings</li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2429"><code>count(distinct)</code></a>
- Native support without falling back to Spark</li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2466"><code>bit_get</code></a>
- Get bit value at position</li>
+</ul>
+<p>New expression capabilities include:</p>
+<ul>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2181">Nested array
literal support</a></li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2425">Array-to-string
cast support</a></li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2472">Spark-compatible
cast from integral to decimal types</a></li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2490">Support for
decimal type to boolean cast</a></li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2316">More date part
expressions</a></li>
+</ul>
+<h3 id="performance-improvements">Performance Improvements<a
class="headerlink" href="#performance-improvements" title="Permanent
link">¶</a></h3>
+<ul>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2417">Improved
BroadcastExchangeExec conversion</a> for better broadcast join
performance</li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2407">Use of
DataFusion's native <code>count_udaf</code></a> instead of
<code>SUM(IF(expr IS NOT NULL, 1, 0))</code></li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2402">New
configuration from shared conf</a> to reduce overhead</li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2579">Buffered index
writes</a> to reduce system calls in shuffle operations</li>
+</ul>
+<h3 id="comet-0110-tpc-h-performance">Comet 0.11.0 TPC-H
Performance<a class="headerlink" href="#comet-0110-tpc-h-performance"
title="Permanent link">¶</a></h3>
+<p>Comet 0.11.0 continues to deliver significant performance
improvements over Spark. In our <a
href="https://github.com/apache/datafusion-comet/pull/2596">TPC-H
benchmarks</a>, Comet reduced overall query runtime from 687 seconds to
302 seconds when processing 100 GB of Parquet data using a single 8-core
executor, achieving a <strong>2.2x speedup</strong>.</p>
+<p><img alt="TPC-H Overall Performance"
src="/images/comet-0.11.0/tpch_allqueries.png"/></p>
+<p>The performance gains are consistent across individual queries, with
most queries showing substantial improvements:</p>
+<p><img alt="TPC-H Query-by-Query Comparison"
src="/images/comet-0.11.0/tpch_queries_compare.png"/></p>
+<p>You can reproduce these benchmarks using our <a
href="https://datafusion.apache.org/comet/contributor-guide/benchmarking.html">Comet
Benchmarking Guide</a>. We encourage you to run your own performance
tests with your workloads.</p>
+<h3 id="apache-iceberg-support">Apache Iceberg Support<a
class="headerlink" href="#apache-iceberg-support" title="Permanent
link">¶</a></h3>
+<ul>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2386">Updated support
for Apache Iceberg 1.9.1</a></li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2442">Additional
Parquet-independent API improvements</a> for Iceberg
integration</li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2510">Improved
resource management</a> in Iceberg reader instances</li>
+</ul>
+<h3 id="ux-improvements">UX Improvements<a class="headerlink"
href="#ux-improvements" title="Permanent link">¶</a></h3>
+<ul>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2412">Added plan
conversion statistics to extended explain info</a> for better
observability</li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2450">Improved
fallback information</a> to help users understand when and why Comet
falls back to Spark</li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2515">Added
<code>backtrace</code> feature</a> to simplify enabling
native backtraces in <code>CometNativeException</code></li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2379">Native log level
is now configurable</a> via Comet configuration</li>
+</ul>
+<h3 id="bug-fixes">Bug Fixes<a class="headerlink" href="#bug-fixes"
title="Permanent link">¶</a></h3>
+<ul>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2398">Resolved issues
with reused broadcast plans in non-AQE mode</a></li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2420">Fixed thread
safety in setNumPartitions</a></li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2440">Improved error
handling when resolving S3 bucket region</a></li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2432">Fixed byte array
literal casting issues</a></li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2438">Corrected
subquery filter pushdown behavior for native_datafusion
scan</a></li>
+</ul>
+<h3 id="documentation-updates">Documentation Updates<a
class="headerlink" href="#documentation-updates" title="Permanent
link">¶</a></h3>
+<ul>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2487">Updated
documentation for native shuffle configuration and tuning</a></li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2496">Added
documentation for ANSI mode support</a> in various functions</li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2474">Improved EC2
benchmarking guide</a></li>
+<li><a
href="https://github.com/apache/datafusion-comet/pull/2568">Split
configuration guide into different sections</a> (scan, exec, shuffle,
etc.) for better organization</li>
+<li>Various clarifications and improvements throughout the
documentation</li>
+</ul>
+<h3 id="spark-compatibility">Spark Compatibility<a class="headerlink"
href="#spark-compatibility" title="Permanent link">¶</a></h3>
+<ul>
+<li>Spark 3.4.3 with JDK 11 &amp; 17, Scala 2.12 &amp;
2.13</li>
+<li>Spark 3.5.4 through 3.5.6 with JDK 11 &amp; 17, Scala 2.12
&amp; 2.13</li>
+<li>Spark 4.0.1 with JDK 17, Scala 2.13</li>
+</ul>
+<p>We are looking for help from the community to fully support Spark
4.0.1. See <a
href="https://github.com/apache/datafusion-comet/issues/1637">EPIC: Support
4.0.0</a> for more information.</p>
+<h2 id="getting-involved">Getting Involved<a class="headerlink"
href="#getting-involved" title="Permanent link">¶</a></h2>
+<p>The Comet project welcomes new contributors. We use the same <a
href="https://datafusion.apache.org/contributor-guide/communication.html#slack-and-discord">Slack
and Discord</a> channels as the main DataFusion
+project and have a weekly <a
href="https://docs.google.com/document/d/1NBpkIAuU7O9h8Br5CbFksDhX-L9TyO9wmGLPMe0Plc8/edit?usp=sharing">DataFusion
video call</a>.</p>
+<p>The easiest way to get involved is to test Comet with your current
Spark jobs and file issues for any bugs or
+performance regressions that you find. See the <a
href="https://datafusion.apache.org/comet/user-guide/installation.html">Getting
Started</a> guide for instructions on downloading and installing
+Comet.</p>
+<p>There are also many <a
href="https://github.com/apache/datafusion-comet/contribute">good first
issues</a> waiting for contributions.</p></content><category
term="blog"></category></entry><entry><title>Apache DataFusion 50.0.0
Released</title><link
href="https://datafusion.apache.org/blog/2025/09/29/datafusion-50.0.0";
rel="alternate"></link><published>2025-09-29T00:00:00+00:00</published><updated>2025-09-29T00:00:00+00:00</updated><author><name>pmc</name></autho
[...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
diff --git a/output/feeds/pmc.rss.xml b/output/feeds/pmc.rss.xml
index 51819d4..194a0ea 100644
--- a/output/feeds/pmc.rss.xml
+++ b/output/feeds/pmc.rss.xml
@@ -1,5 +1,27 @@
<?xml version="1.0" encoding="utf-8"?>
-<rss version="2.0"><channel><title>Apache DataFusion Blog -
pmc</title><link>https://datafusion.apache.org/blog/</link><description></description><lastBuildDate>Mon,
29 Sep 2025 00:00:00 +0000</lastBuildDate><item><title>Apache DataFusion
50.0.0
Released</title><link>https://datafusion.apache.org/blog/2025/09/29/datafusion-50.0.0</link><description><!--
+<rss version="2.0"><channel><title>Apache DataFusion Blog -
pmc</title><link>https://datafusion.apache.org/blog/</link><description></description><lastBuildDate>Tue,
21 Oct 2025 00:00:00 +0000</lastBuildDate><item><title>Apache DataFusion Comet
0.11.0
Release</title><link>https://datafusion.apache.org/blog/2025/10/21/datafusion-comet-0.11.0</link><description><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<p>The Apache DataFusion PMC is pleased to announce version 0.11.0 of
the <a href="https://datafusion.apache.org/comet/">Comet</a>
subproject.</p>
+<p>Comet is an accelerator for Apache Spark that translates Spark
physical plans to DataFusion physical plans for
+improved performance and efficiency without requiring any code
changes.</p>
+<p>This release covers approximately five weeks of development
…</p></description><dc:creator
xmlns:dc="http://purl.org/dc/elements/1.1/";>pmc</dc:creator><pubDate>Tue, 21
Oct 2025 00:00:00 +0000</pubDate><guid
isPermaLink="false">tag:datafusion.apache.org,2025-10-21:/blog/2025/10/21/datafusion-comet-0.11.0</guid><category>blog</category></item><item><title>Apache
DataFusion 50.0.0
Released</title><link>https://datafusion.apache.org/blog/2025/09/29/datafusion-50.0.0</link><des
[...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
diff --git a/output/images/comet-0.11.0/tpch_allqueries.png
b/output/images/comet-0.11.0/tpch_allqueries.png
new file mode 100644
index 0000000..7daabac
Binary files /dev/null and b/output/images/comet-0.11.0/tpch_allqueries.png
differ
diff --git a/output/images/comet-0.11.0/tpch_queries_compare.png
b/output/images/comet-0.11.0/tpch_queries_compare.png
new file mode 100644
index 0000000..bd8a589
Binary files /dev/null and
b/output/images/comet-0.11.0/tpch_queries_compare.png differ
diff --git a/output/index.html b/output/index.html
index b38853e..880f8b0 100644
--- a/output/index.html
+++ b/output/index.html
@@ -45,6 +45,47 @@
<p><i>Here you can find the latest updates from DataFusion and
related projects.</i></p>
+ <!-- Post -->
+ <div class="row">
+ <div class="callout">
+ <article class="post">
+ <header>
+ <div class="title">
+ <h1><a
href="/blog/2025/10/21/datafusion-comet-0.11.0">Apache DataFusion Comet 0.11.0
Release</a></h1>
+ <p>Posted on: Tue 21 October 2025 by pmc</p>
+ <p><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<p>The Apache DataFusion PMC is pleased to announce version 0.11.0 of the <a
href="https://datafusion.apache.org/comet/";>Comet</a> subproject.</p>
+<p>Comet is an accelerator for Apache Spark that translates Spark physical
plans to DataFusion physical plans for
+improved performance and efficiency without requiring any code changes.</p>
+<p>This release covers approximately five weeks of development …</p></p>
+ <footer>
+ <ul class="actions">
+ <div style="text-align: right"><a
href="/blog/2025/10/21/datafusion-comet-0.11.0" class="button medium">Continue
Reading</a></div>
+ </ul>
+ <ul class="stats">
+ </ul>
+ </footer>
+ </article>
+ </div>
+ </div>
<!-- Post -->
<div class="row">
<div class="callout">
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
