This is an automated email from the ASF dual-hosted git repository.
MaxGekk pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new a81e3ebe0bb7 [SPARK-57579][PYTHON] Add PySpark support for unix_nanos
function
a81e3ebe0bb7 is described below
commit a81e3ebe0bb7872688ecb9abdafae865b505a145
Author: Jubin Soni <[email protected]>
AuthorDate: Wed Jun 24 14:06:27 2026 +0200
[SPARK-57579][PYTHON] Add PySpark support for unix_nanos function
## What is the purpose of the change
Fixes SPARK-57579 (follow-up to SPARK-57527) — adds `unix_nanos` to the
PySpark API (`pyspark.sql.functions` and PySpark Connect), completing the
epoch-unit function family in Python.
The SQL function and Scala API were added in SPARK-57527, but Python
support was explicitly deferred. The full family is:
| Function | PySpark before this PR | PySpark after this PR |
|---|---|---|
| `unix_seconds` | present | present |
| `unix_millis` | present | present |
| `unix_micros` | present | present |
| `unix_nanos` | **missing** | **added** |
The gap was acknowledged in the parity test (`expected_missing_in_py`) with
a comment pointing to this follow-up.
## Brief change log
- `python/pyspark/sql/functions/builtin.py`: added `unix_nanos(col)` after
`unix_micros`, decorated with `_try_remote_functions`, with full docstring
(`versionadded:: 4.3.0`, parameters, return type, See Also links, and two
doctests covering a nanosecond-precision `TIMESTAMP_NTZ` input and a `NULL`
input)
- `python/pyspark/sql/functions/__init__.py`: exported `unix_nanos` in
alphabetical order between `unix_millis` and `unix_seconds`
- `python/pyspark/sql/connect/functions/builtin.py`: added Connect-side
wrapper for `unix_nanos` inheriting its docstring from the main function,
following the same pattern as `unix_micros`
- `python/pyspark/sql/tests/test_functions.py`: removed `"unix_nanos"` from
`expected_missing_in_py` (set is now empty)
## Verifying this change
This change is covered by the existing parity test in `FunctionsTestsMixin`:
- `test_function_parity`: previously allowlisted `unix_nanos` as an
expected gap; removing it from `expected_missing_in_py` means the test will now
fail if `unix_nanos` is ever missing from the Python API again
- The two doctests in the `unix_nanos` docstring verify:
- A nanosecond-precision `TIMESTAMP_NTZ` input returns the correct
`DECIMAL(21, 0)` nanosecond count
- A `NULL` input returns `NULL`
## Does this pull request potentially affect one of the following parts
- Dependencies (does it add or upgrade a dependency): **no**
- The public API, i.e., is any changed class annotated with
`Public`/`Evolving`: **yes** — `unix_nanos` is a new public PySpark function
- The serializers: **no**
- The runtime per-record code paths (performance sensitive): **no** — this
is a Python wrapper only; the JVM expression `UnixNanos` is unchanged
- Anything that affects deployment or recovery: **no**
- The S3 file system connector: **no**
## Documentation
Does this pull request introduce a new feature? **yes** —
`pyspark.sql.functions.unix_nanos` is a new public API
If yes, how is the feature documented? inline docstring with parameter
description, return type, See Also links, and doctests in `builtin.py`
## Was generative AI tooling used to co-author this PR?
- [x] Yes — Claude Code was used as a pair-programming assistant. All code
was written, understood, and verified by the author.
Generated-by: Claude Sonnet 4.8
Closes #56626 from jubins/j-SPARK-57579-unix-nanos-pyspark-support.
Authored-by: Jubin Soni <[email protected]>
Signed-off-by: Max Gekk <[email protected]>
(cherry picked from commit 7c66ffc895a56f3668700ab2d88d13a7d5bfab5b)
Signed-off-by: Max Gekk <[email protected]>
---
.../source/reference/pyspark.sql/functions.rst | 1 +
python/pyspark/sql/connect/functions/builtin.py | 7 +++
python/pyspark/sql/functions/__init__.py | 1 +
python/pyspark/sql/functions/builtin.py | 51 ++++++++++++++++++++++
python/pyspark/sql/tests/test_functions.py | 5 +--
5 files changed, 62 insertions(+), 3 deletions(-)
diff --git a/python/docs/source/reference/pyspark.sql/functions.rst
b/python/docs/source/reference/pyspark.sql/functions.rst
index e41066256bd4..3ad3ae9cdf12 100644
--- a/python/docs/source/reference/pyspark.sql/functions.rst
+++ b/python/docs/source/reference/pyspark.sql/functions.rst
@@ -327,6 +327,7 @@ Date and Timestamp Functions
unix_date
unix_micros
unix_millis
+ unix_nanos
unix_seconds
unix_timestamp
weekday
diff --git a/python/pyspark/sql/connect/functions/builtin.py
b/python/pyspark/sql/connect/functions/builtin.py
index 828ac151cd8c..00183fe283f0 100644
--- a/python/pyspark/sql/connect/functions/builtin.py
+++ b/python/pyspark/sql/connect/functions/builtin.py
@@ -3531,6 +3531,13 @@ def unix_millis(col: "ColumnOrName") -> Column:
unix_millis.__doc__ = pysparkfuncs.unix_millis.__doc__
+def unix_nanos(col: "ColumnOrName") -> Column:
+ return _invoke_function_over_columns("unix_nanos", col)
+
+
+unix_nanos.__doc__ = pysparkfuncs.unix_nanos.__doc__
+
+
def unix_seconds(col: "ColumnOrName") -> Column:
return _invoke_function_over_columns("unix_seconds", col)
diff --git a/python/pyspark/sql/functions/__init__.py
b/python/pyspark/sql/functions/__init__.py
index 6d1f794474eb..914b9c7fbcb7 100644
--- a/python/pyspark/sql/functions/__init__.py
+++ b/python/pyspark/sql/functions/__init__.py
@@ -276,6 +276,7 @@ __all__ = [ # noqa: F405
"unix_date",
"unix_micros",
"unix_millis",
+ "unix_nanos",
"unix_seconds",
"unix_timestamp",
"weekday",
diff --git a/python/pyspark/sql/functions/builtin.py
b/python/pyspark/sql/functions/builtin.py
index 43340405193f..148e6d5184cf 100644
--- a/python/pyspark/sql/functions/builtin.py
+++ b/python/pyspark/sql/functions/builtin.py
@@ -11792,6 +11792,57 @@ def unix_millis(col: "ColumnOrName") -> Column:
return _invoke_function_over_columns("unix_millis", col)
+@_try_remote_functions
+def unix_nanos(col: "ColumnOrName") -> Column:
+ """Returns the number of nanoseconds since 1970-01-01 00:00:00 UTC as
``DECIMAL(21, 0)``.
+ Only supported for ``TIMESTAMP_LTZ(p)`` and ``TIMESTAMP_NTZ(p)`` with
precision ``p``
+ in ``[7, 9]``.
+
+ .. versionadded:: 4.3.0
+
+ Parameters
+ ----------
+ col : :class:`~pyspark.sql.Column` or column name
+ input column of nanosecond-precision timestamp values to convert.
+
+ Returns
+ -------
+ :class:`~pyspark.sql.Column`
+ the number of nanoseconds since 1970-01-01 00:00:00 UTC as
``DECIMAL(21, 0)``.
+
+ See Also
+ --------
+ :meth:`pyspark.sql.functions.unix_date`
+ :meth:`pyspark.sql.functions.unix_seconds`
+ :meth:`pyspark.sql.functions.unix_millis`
+ :meth:`pyspark.sql.functions.unix_micros`
+
+ Examples
+ --------
+ >>> import pyspark.sql.functions as sf
+ >>> spark.conf.set("spark.sql.timestampNanosTypes.enabled", "true")
+ >>> df = spark.sql(
+ ... "SELECT TIMESTAMP_NTZ '2020-01-01 13:24:35.123456789' AS ts"
+ ... )
+ >>> df.select('*', sf.unix_nanos('ts')).show(truncate=False)
+ +-----------------------------+-------------------+
+ |ts |unix_nanos(ts) |
+ +-----------------------------+-------------------+
+ |2020-01-01 13:24:35.123456789|1577885075123456789|
+ +-----------------------------+-------------------+
+
+ >>> df.select(sf.unix_nanos(sf.lit(None).cast('timestamp_ntz(9)'))).show()
+ +------------------------------------------+
+ |unix_nanos(CAST(NULL AS TIMESTAMP_NTZ(9)))|
+ +------------------------------------------+
+ | NULL|
+ +------------------------------------------+
+
+ >>> spark.conf.unset("spark.sql.timestampNanosTypes.enabled")
+ """
+ return _invoke_function_over_columns("unix_nanos", col)
+
+
@_try_remote_functions
def unix_seconds(col: "ColumnOrName") -> Column:
"""Returns the number of seconds since 1970-01-01 00:00:00 UTC.
diff --git a/python/pyspark/sql/tests/test_functions.py
b/python/pyspark/sql/tests/test_functions.py
index c9ca0fca96a7..16928193db6d 100644
--- a/python/pyspark/sql/tests/test_functions.py
+++ b/python/pyspark/sql/tests/test_functions.py
@@ -83,9 +83,8 @@ class FunctionsTestsMixin:
# Functions that we expect to be missing in python until they are
added to pyspark
expected_missing_in_py = {
- "unix_nanos", # SPARK-57527: PySpark support tracked as a
follow-up
- "timestamp_nanos", # SPARK-57526: PySpark support tracked as a
follow-up
- }
+ "timestamp_nanos"
+ } # SPARK-57526: PySpark support tracked as a follow-up
self.assertEqual(
expected_missing_in_py, missing_in_py, "Missing functions in
pyspark not as expected"
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
