andygrove opened a new pull request, #6778:
URL: https://github.com/apache/datafusion-comet/pull/6778
## Which issue does this PR close?
No issue. The answer records what the prototype in #6697 found for #6694.
## Rationale for this change
With the Rust UDF API (#4459) documented, an obvious question is whether
Java and Scala get an equivalent. #6697 prototyped one, and the answer is no
for now. Once some overheads in the code generator were removed, the vectorized
form of a simple function saved about 3 ns per row, about a fifth of a query
that does little besides scan one column. That does not justify asking users to
rewrite their functions against Comet's relocated Arrow classes and rebuild
them for every Comet release. An FAQ entry records the answer and the
reasoning, so it doesn't have to be worked out again.
## What changes are included in this PR?
- A new page, `about/faq.md`, titled "Frequently Asked Questions", with one
question so far: "Does Comet plan to add an API for vectorized Java/Scala UDFs
similar to the Rust UDF API?" The answer:
- says that ordinary Scala and Java UDFs already run in Comet's pipeline
with no code changes
- summarizes what the prototype measured and what the API would cost users
- says where a vectorized function could still pay off, points to #6694
for those use cases, and points to the Rust UDF API for native speed
- The About page lists the new page and adds it to its table of contents.
The figures come from `CometJvmUdfBenchmark` in #6697, measured with #6715,
which runs Spark's null check around a Scala UDF inside the code generator's
kernel.
## How are these changes tested?
Documentation only. `prettier --check` passes on both files. The docs site
builds only on pushes to main, so the page has not been rendered.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
