kosiew commented on code in PR #115:
URL: https://github.com/apache/datafusion-spark/pull/115#discussion_r4627218322
##########
dev/update_function_docs.sh:
##########
@@ -0,0 +1,40 @@
+#!/usr/bin/env bash
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: See the NOTICE file distributed with this work for
additional information regarding copyright ownership
+# SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF) under
one or more contributor license agreements
+
+# Regenerates docs/functions.md from the #[user_doc] attributes on each
function.
+#
+# Usage: dev/update_function_docs.sh
+
+set -euo pipefail
+
+ROOT_DIR="$(git rev-parse --show-toplevel)"
+cd "${ROOT_DIR}"
+
+TARGET_FILE="docs/functions.md"
+mkdir -p "$(dirname "${TARGET_FILE}")"
+
+{
+ cat <<'HEADER'
+<!--
+SPDX-License-Identifier: Apache-2.0
+SPDX-FileCopyrightText: See the NOTICE file distributed with this work for
additional information regarding copyright ownership
+SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF) under
one or more contributor license agreements
+-->
+
+<!---
+This file was generated by the dev/update_function_docs.sh script.
+Do not edit it manually as changes will be overwritten.
+Instead, edit the #[user_doc] attributes on the functions and run the script.
+-->
+
+# datafusion-spark Function Reference
+
+Functions in this crate emulate Apache Spark's behavior. Each entry describes
how the function
+differs from the DataFusion built-in of the same name, if there is one.
+HEADER
+ cargo run --quiet -p datafusion-spark-docgen --bin print_functions_docs
+} > "${TARGET_FILE}"
Review Comment:
A failed `cargo run` leaves the reference truncated to its header. Could we
generate into a temporary file, replace the target only on success, and clean
up with a trap?
##########
crates/spark/README.md:
##########
@@ -24,7 +24,8 @@ errors, ...). This crate is useful when you are:
- running SQL migrated from Spark on DataFusion.
Not every Spark function is implemented yet. Progress is tracked in the
-[`datafusion-spark` epic]; see the [API documentation] for the functions
available today.
+[`datafusion-spark` epic]. See the [function reference] for the functions
available today, with
Review Comment:
The reference excludes registered functions without `documentation()`. Could
we say "documented functions" here and in `README.md:52` and
`crates/spark/src/lib.rs:49` to avoid overstating coverage?
##########
.github/workflows/lint.yml:
##########
@@ -51,3 +51,18 @@ jobs:
run: cargo check --no-default-features -p datafusion-spark
- name: Check datafusion-spark (core)
run: cargo check --no-default-features --features core -p
datafusion-spark
+
+ function-docs:
+ name: function docs up to date
+ runs-on: ubuntu-latest
+ steps:
+ - uses: actions/checkout@v4
+ - run: rustup toolchain install stable --profile minimal
--no-self-update && rustup default stable
+ - uses: Swatinem/rust-cache@v2
+ - run: dev/update_function_docs.sh
+ - name: Check docs/functions.md is up to date
+ run: |
+ if ! git diff --exit-code -- docs/functions.md; then
Review Comment:
Could we use `git status --porcelain -- docs/functions.md` instead? `git
diff --exit-code` misses untracked files, so CI passes if the reference is
deleted and regenerated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]