This is an automated email from the ASF dual-hosted git repository.
MaxGekk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 526d88da65e0 [SPARK-57666][SQL] Add Timestamp nano benchmarks
526d88da65e0 is described below
commit 526d88da65e0ba200fcca80285733731f4d3e2ba
Author: Stevo Mitric <[email protected]>
AuthorDate: Thu Jun 25 18:03:36 2026 +0200
[SPARK-57666][SQL] Add Timestamp nano benchmarks
### What changes were proposed in this pull request?
Add `TimestampNanosBenchmark`, comparing the microsecond timestamp path
against the nanosecond timestamp types `TIMESTAMP_NTZ(p)` / `TIMESTAMP_LTZ(p)`
(p in [7, 9]) across cast, string parse/format, hashing, sort, group-by,
MIN/MAX, datetime functions, and Parquet/ORC read/write.
### Why are the changes needed?
To quantify the per-row overhead of the 16-byte `TimestampNanosVal` carrier
vs the 8-byte micros `Long`, and to surface known nanos gaps (no radix sort
prefix; row-based-only vectorized Parquet read).
### Does this PR introduce _any_ user-facing change?
No, test-only.
### How was this patch tested?
N/A (benchmark). Results regenerated via the `Run benchmarks` GitHub Action.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 4.8)
Closes #56744 from stevomitric/stevomitric/timestamp-nanos-benchmark.
Authored-by: Stevo Mitric <[email protected]>
Signed-off-by: Max Gekk <[email protected]>
---
.../benchmarks/TimestampNanosBenchmark-results.txt | 294 +++++++++++++++++++++
.../benchmark/TimestampNanosBenchmark.scala | 245 +++++++++++++++++
2 files changed, 539 insertions(+)
diff --git a/sql/core/benchmarks/TimestampNanosBenchmark-results.txt
b/sql/core/benchmarks/TimestampNanosBenchmark-results.txt
new file mode 100644
index 000000000000..33a104b9a537
--- /dev/null
+++ b/sql/core/benchmarks/TimestampNanosBenchmark-results.txt
@@ -0,0 +1,294 @@
+================================================================================================
+Cast
+================================================================================================
+
+OpenJDK 64-Bit Server VM 17.0.15+6-Ubuntu-0ubuntu120.04 on Linux
5.4.0-1160-aws-fips
+Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz
+micros -> string: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+micros -> string wholestage off 3586 3610
35 2.8 358.6 1.0X
+micros -> string wholestage on 3663 3682
27 2.7 366.3 1.0X
+
+OpenJDK 64-Bit Server VM 17.0.15+6-Ubuntu-0ubuntu120.04 on Linux
5.4.0-1160-aws-fips
+Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz
+nanos(9) -> string: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+nanos(9) -> string wholestage off 8024 8048
33 1.2 802.4 1.0X
+nanos(9) -> string wholestage on 7987 8002
10 1.3 798.7 1.0X
+
+OpenJDK 64-Bit Server VM 17.0.15+6-Ubuntu-0ubuntu120.04 on Linux
5.4.0-1160-aws-fips
+Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz
+nanos(7) -> string: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+nanos(7) -> string wholestage off 7879 7884
7 1.3 787.9 1.0X
+nanos(7) -> string wholestage on 7882 7936
70 1.3 788.2 1.0X
+
+OpenJDK 64-Bit Server VM 17.0.15+6-Ubuntu-0ubuntu120.04 on Linux
5.4.0-1160-aws-fips
+Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz
+micros -> nanos(9): Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+micros -> nanos(9) wholestage off 868 870
4 11.5 86.8 1.0X
+micros -> nanos(9) wholestage on 838 845
12 11.9 83.8 1.0X
+
+OpenJDK 64-Bit Server VM 17.0.15+6-Ubuntu-0ubuntu120.04 on Linux
5.4.0-1160-aws-fips
+Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz
+nanos(9) -> micros: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+nanos(9) -> micros wholestage off 1999 2024
35 5.0 199.9 1.0X
+nanos(9) -> micros wholestage on 1988 2025
57 5.0 198.8 1.0X
+
+OpenJDK 64-Bit Server VM 17.0.15+6-Ubuntu-0ubuntu120.04 on Linux
5.4.0-1160-aws-fips
+Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz
+nanos(9) -> nanos(7): Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+nanos(9) -> nanos(7) wholestage off 2042 2044
3 4.9 204.2 1.0X
+nanos(9) -> nanos(7) wholestage on 2024 2031
4 4.9 202.4 1.0X
+
+OpenJDK 64-Bit Server VM 17.0.15+6-Ubuntu-0ubuntu120.04 on Linux
5.4.0-1160-aws-fips
+Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz
+micros -> date: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+micros -> date wholestage off 1418 1515
136 7.0 141.8 1.0X
+micros -> date wholestage on 1344 1359
18 7.4 134.4 1.1X
+
+OpenJDK 64-Bit Server VM 17.0.15+6-Ubuntu-0ubuntu120.04 on Linux
5.4.0-1160-aws-fips
+Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz
+nanos(9) -> date: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+nanos(9) -> date wholestage off 2506 2513
10 4.0 250.6 1.0X
+nanos(9) -> date wholestage on 2525 2542
21 4.0 252.5 1.0X
+
+
+================================================================================================
+Parse from string
+================================================================================================
+
+OpenJDK 64-Bit Server VM 17.0.15+6-Ubuntu-0ubuntu120.04 on Linux
5.4.0-1160-aws-fips
+Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz
+string(6 digits) -> micros: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+-------------------------------------------------------------------------------------------------------------------------
+string(6 digits) -> micros wholestage off 288 291
4 3.5 288.1 1.0X
+string(6 digits) -> micros wholestage on 279 282
3 3.6 278.9 1.0X
+
+OpenJDK 64-Bit Server VM 17.0.15+6-Ubuntu-0ubuntu120.04 on Linux
5.4.0-1160-aws-fips
+Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz
+string(9 digits) -> nanos(9): Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+---------------------------------------------------------------------------------------------------------------------------
+string(9 digits) -> nanos(9) wholestage off 313 327
19 3.2 313.2 1.0X
+string(9 digits) -> nanos(9) wholestage on 305 310
9 3.3 305.1 1.0X
+
+OpenJDK 64-Bit Server VM 17.0.15+6-Ubuntu-0ubuntu120.04 on Linux
5.4.0-1160-aws-fips
+Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz
+string(7 digits) -> nanos(7): Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+---------------------------------------------------------------------------------------------------------------------------
+string(7 digits) -> nanos(7) wholestage off 295 301
9 3.4 295.1 1.0X
+string(7 digits) -> nanos(7) wholestage on 293 295
2 3.4 292.7 1.0X
+
+
+================================================================================================
+Hashing
+================================================================================================
+
+OpenJDK 64-Bit Server VM 17.0.15+6-Ubuntu-0ubuntu120.04 on Linux
5.4.0-1160-aws-fips
+Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz
+murmur3 hash(micros): Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+murmur3 hash(micros) wholestage off 858 868
14 11.7 85.8 1.0X
+murmur3 hash(micros) wholestage on 834 836
3 12.0 83.4 1.0X
+
+OpenJDK 64-Bit Server VM 17.0.15+6-Ubuntu-0ubuntu120.04 on Linux
5.4.0-1160-aws-fips
+Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz
+murmur3 hash(nanos(9)): Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+murmur3 hash(nanos(9)) wholestage off 1987 2044
80 5.0 198.7 1.0X
+murmur3 hash(nanos(9)) wholestage on 2024 2030
6 4.9 202.4 1.0X
+
+OpenJDK 64-Bit Server VM 17.0.15+6-Ubuntu-0ubuntu120.04 on Linux
5.4.0-1160-aws-fips
+Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz
+xxhash64(micros): Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+xxhash64(micros) wholestage off 806 807
1 12.4 80.6 1.0X
+xxhash64(micros) wholestage on 801 808
8 12.5 80.1 1.0X
+
+OpenJDK 64-Bit Server VM 17.0.15+6-Ubuntu-0ubuntu120.04 on Linux
5.4.0-1160-aws-fips
+Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz
+xxhash64(nanos(9)): Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+xxhash64(nanos(9)) wholestage off 2043 2043
0 4.9 204.3 1.0X
+xxhash64(nanos(9)) wholestage on 2003 2038
32 5.0 200.3 1.0X
+
+
+================================================================================================
+Sort (global ORDER BY)
+================================================================================================
+
+OpenJDK 64-Bit Server VM 17.0.15+6-Ubuntu-0ubuntu120.04 on Linux
5.4.0-1160-aws-fips
+Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz
+order by micros: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+order by micros wholestage off 2345 2399
77 4.3 234.5 1.0X
+order by micros wholestage on 2325 2344
22 4.3 232.5 1.0X
+
+OpenJDK 64-Bit Server VM 17.0.15+6-Ubuntu-0ubuntu120.04 on Linux
5.4.0-1160-aws-fips
+Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz
+order by nanos(9): Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+order by nanos(9) wholestage off 3037 3172
190 3.3 303.7 1.0X
+order by nanos(9) wholestage on 3103 3186
65 3.2 310.3 1.0X
+
+
+================================================================================================
+Group-by aggregation (~1M distinct keys)
+================================================================================================
+
+OpenJDK 64-Bit Server VM 17.0.15+6-Ubuntu-0ubuntu120.04 on Linux
5.4.0-1160-aws-fips
+Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz
+group by micros key: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+group by micros key wholestage off 2080 2114
47 4.8 208.0 1.0X
+group by micros key wholestage on 1888 1919
19 5.3 188.8 1.1X
+
+OpenJDK 64-Bit Server VM 17.0.15+6-Ubuntu-0ubuntu120.04 on Linux
5.4.0-1160-aws-fips
+Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz
+group by nanos(9) key: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+group by nanos(9) key wholestage off 3576 3611
50 2.8 357.6 1.0X
+group by nanos(9) key wholestage on 3039 3127
82 3.3 303.9 1.2X
+
+
+================================================================================================
+MIN/MAX aggregation
+================================================================================================
+
+OpenJDK 64-Bit Server VM 17.0.15+6-Ubuntu-0ubuntu120.04 on Linux
5.4.0-1160-aws-fips
+Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz
+min/max micros: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+min/max micros wholestage off 998 1004
7 10.0 99.8 1.0X
+min/max micros wholestage on 617 651
33 16.2 61.7 1.6X
+
+OpenJDK 64-Bit Server VM 17.0.15+6-Ubuntu-0ubuntu120.04 on Linux
5.4.0-1160-aws-fips
+Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz
+min/max nanos(9): Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+min/max nanos(9) wholestage off 2402 2410
11 4.2 240.2 1.0X
+min/max nanos(9) wholestage on 1895 1912
26 5.3 189.5 1.3X
+
+
+================================================================================================
+Datetime functions
+================================================================================================
+
+OpenJDK 64-Bit Server VM 17.0.15+6-Ubuntu-0ubuntu120.04 on Linux
5.4.0-1160-aws-fips
+Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz
+hour(micros): Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+hour(micros) wholestage off 749 758
13 13.4 74.9 1.0X
+hour(micros) wholestage on 723 728
5 13.8 72.3 1.0X
+
+OpenJDK 64-Bit Server VM 17.0.15+6-Ubuntu-0ubuntu120.04 on Linux
5.4.0-1160-aws-fips
+Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz
+hour(nanos(9)): Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+hour(nanos(9)) wholestage off 1803 1816
19 5.5 180.3 1.0X
+hour(nanos(9)) wholestage on 1765 1789
34 5.7 176.5 1.0X
+
+OpenJDK 64-Bit Server VM 17.0.15+6-Ubuntu-0ubuntu120.04 on Linux
5.4.0-1160-aws-fips
+Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz
+second(micros): Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+second(micros) wholestage off 777 783
9 12.9 77.7 1.0X
+second(micros) wholestage on 863 1117
146 11.6 86.3 0.9X
+
+OpenJDK 64-Bit Server VM 17.0.15+6-Ubuntu-0ubuntu120.04 on Linux
5.4.0-1160-aws-fips
+Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz
+second(nanos(9)): Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+second(nanos(9)) wholestage off 2094 2098
6 4.8 209.4 1.0X
+second(nanos(9)) wholestage on 1781 2059
340 5.6 178.1 1.2X
+
+OpenJDK 64-Bit Server VM 17.0.15+6-Ubuntu-0ubuntu120.04 on Linux
5.4.0-1160-aws-fips
+Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz
+extract(SECOND, micros): Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+extract(SECOND, micros) wholestage off 880 880
0 11.4 88.0 1.0X
+extract(SECOND, micros) wholestage on 993 1215
145 10.1 99.3 0.9X
+
+OpenJDK 64-Bit Server VM 17.0.15+6-Ubuntu-0ubuntu120.04 on Linux
5.4.0-1160-aws-fips
+Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz
+extract(SECOND, nanos(9)): Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+extract(SECOND, nanos(9)) wholestage off 2409 2470
87 4.2 240.9 1.0X
+extract(SECOND, nanos(9)) wholestage on 2217 2346
132 4.5 221.7 1.1X
+
+OpenJDK 64-Bit Server VM 17.0.15+6-Ubuntu-0ubuntu120.04 on Linux
5.4.0-1160-aws-fips
+Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz
+unix_micros(micros): Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+unix_micros(micros) wholestage off 295 301
8 33.8 29.5 1.0X
+unix_micros(micros) wholestage on 254 272
18 39.4 25.4 1.2X
+
+OpenJDK 64-Bit Server VM 17.0.15+6-Ubuntu-0ubuntu120.04 on Linux
5.4.0-1160-aws-fips
+Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz
+unix_nanos(nanos(9)): Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+unix_nanos(nanos(9)) wholestage off 2450 2478
40 4.1 245.0 1.0X
+unix_nanos(nanos(9)) wholestage on 2398 2450
59 4.2 239.8 1.0X
+
+OpenJDK 64-Bit Server VM 17.0.15+6-Ubuntu-0ubuntu120.04 on Linux
5.4.0-1160-aws-fips
+Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz
+micros + interval 1 hour: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+micros + interval 1 hour wholestage off 1727 1763
51 5.8 172.7 1.0X
+micros + interval 1 hour wholestage on 1693 1750
52 5.9 169.3 1.0X
+
+OpenJDK 64-Bit Server VM 17.0.15+6-Ubuntu-0ubuntu120.04 on Linux
5.4.0-1160-aws-fips
+Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz
+nanos(9) + interval 1 hour: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+-------------------------------------------------------------------------------------------------------------------------
+nanos(9) + interval 1 hour wholestage off 3678 3709
44 2.7 367.8 1.0X
+nanos(9) + interval 1 hour wholestage on 3192 3300
89 3.1 319.2 1.2X
+
+
+================================================================================================
+Parquet write/read
+================================================================================================
+
+OpenJDK 64-Bit Server VM 17.0.15+6-Ubuntu-0ubuntu120.04 on Linux
5.4.0-1160-aws-fips
+Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz
+Save to Parquet: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+micros (TIMESTAMP_NTZ) 2279 2279
0 4.4 227.9 1.0X
+nanos(9) (TIMESTAMP_NTZ(9)) 3485 3485
0 2.9 348.5 0.7X
+
+OpenJDK 64-Bit Server VM 17.0.15+6-Ubuntu-0ubuntu120.04 on Linux
5.4.0-1160-aws-fips
+Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz
+Load from Parquet: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+micros, vectorized on 532 616
76 18.8 53.2 1.0X
+micros, vectorized off 1798 2067
447 5.6 179.8 0.3X
+nanos(9), vectorized on (row-based) 1949 1982
29 5.1 194.9 0.3X
+nanos(9), vectorized off (row-based) 1804 1817
12 5.5 180.4 0.3X
+
+
+================================================================================================
+ORC write/read
+================================================================================================
+
+OpenJDK 64-Bit Server VM 17.0.15+6-Ubuntu-0ubuntu120.04 on Linux
5.4.0-1160-aws-fips
+Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz
+Save to ORC: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+micros (TIMESTAMP_NTZ) 1257 1257
0 8.0 125.7 1.0X
+nanos(9) (TIMESTAMP_NTZ(9)) 5537 5537
0 1.8 553.7 0.2X
+
+OpenJDK 64-Bit Server VM 17.0.15+6-Ubuntu-0ubuntu120.04 on Linux
5.4.0-1160-aws-fips
+Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz
+Load from ORC: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+micros, vectorized on 361 398
42 27.7 36.1 1.0X
+micros, vectorized off 1026 1047
19 9.7 102.6 0.4X
+nanos(9), vectorized on 1504 1517
16 6.7 150.4 0.2X
+nanos(9), vectorized off 2213 2264
44 4.5 221.3 0.2X
+
+
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/TimestampNanosBenchmark.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/TimestampNanosBenchmark.scala
new file mode 100644
index 000000000000..97372d8402a9
--- /dev/null
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/TimestampNanosBenchmark.scala
@@ -0,0 +1,245 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.benchmark
+
+import org.apache.spark.benchmark.Benchmark
+import
org.apache.spark.sql.catalyst.util.DateTimeTestUtils.{withDefaultTimeZone, LA}
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Synthetic benchmark comparing the microsecond timestamp path against the
nanosecond-precision
+ * timestamp types `TIMESTAMP_NTZ(p)` / `TIMESTAMP_LTZ(p)`, `p in [7, 9]`
(umbrella SPARK-56822).
+ *
+ * Most groups pair the same operation over a microsecond timestamp and the
nanosecond type, so the
+ * per-row delta of the nanos path (a 16-byte `TimestampNanosVal` =
epochMicros:Long +
+ * nanosWithinMicro:Short carrier, vs a plain 8-byte Long) is read directly
off adjacent rows. A few
+ * groups pair related operations instead: Cast includes the cross-casts
(micros <-> nanos(9)) and a
+ * nanos-only narrowing (nanos(9) -> nanos(7)), and Datetime functions pairs
`unix_micros` with the
+ * nanos-only `unix_nanos`.
+ * The areas where nanos is expected to diverge from micros:
+ * - hashing: two hash ops (hashInt(nanos), hashLong(micros)) vs one;
+ * - ordering/sort: nanos has NO radix sort prefix, so it falls back to a
full `compareTo` per
+ * comparison (micros sort on the primitive Long prefix);
+ * - Parquet read: nanos read is row-based only (vectorized read is disabled
for the 16-byte
+ * two-field carrier), whereas micros read is vectorized;
+ * - ORC read: nanos reads vectorized over the full year range, like micros.
+ *
+ * Parsing and cast-to-string are additionally spot-checked at p=7 (alongside
p=9): the 16-byte
+ * carrier is identical for all p, so only fractional-digit
truncation/formatting differs.
+ *
+ * The nanosecond timestamp types are behind the preview flag
+ * `spark.sql.timestampNanosTypes.enabled` (default = `isTesting`).
`BenchmarkBase.main` sets
+ * `IS_TESTING=true`, so the flag is on during a benchmark run; this suite
also sets it explicitly.
+ *
+ * To run this benchmark:
+ * {{{
+ * 1. without sbt:
+ * bin/spark-submit --class <this class>
+ * --jars <spark core test jar>,<spark catalyst test jar> <sql core
test jar>
+ * 2. build/sbt "sql/Test/runMain <this class>"
+ * 3. generate result:
+ * SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt "sql/Test/runMain <this
class>"
+ * Results will be written to
"benchmarks/TimestampNanosBenchmark-results.txt".
+ * }}}
+ */
+object TimestampNanosBenchmark extends SqlBasedBenchmark {
+
+ private val N = 10000000
+
+ //
---------------------------------------------------------------------------
+ // Value generators (SQL expressions over `id` from spark.range).
+ //
+ // Nanos values are built as `id` seconds + (id % 1000)
nanoseconds-of-microsecond, so both the
+ // micros field and the sub-microsecond field vary across rows. The whole
range stays in ~1970,
+ // inside the Parquet INT64-NANOS window (1677-09-21 .. 2262-04-11) so
Parquet writes never throw
+ // DATETIME_OVERFLOW.
+ //
---------------------------------------------------------------------------
+ private val microLtz = "timestamp_seconds(id)"
+ private val microNtz = "cast(timestamp_seconds(id) as timestamp_ntz)"
+ private val nanosArg = "id * 1000000000 + id % 1000"
+ private val nanosLtz = s"timestamp_nanos($nanosArg)"
+ private val nanosNtz = s"cast(timestamp_nanos($nanosArg) as
timestamp_ntz(9))"
+ private val nanosNtz7 = s"cast(timestamp_nanos($nanosArg) as
timestamp_ntz(7))"
+
+ // 9-, 7- and 6-digit fractional-second strings for the parsing group.
+ private val nanosStr =
+ "concat('2019-01-27 11:02:01.', lpad(cast(id % 1000000000 as string), 9,
'0'))"
+ private val nanosStr7 =
+ "concat('2019-01-27 11:02:01.', lpad(cast(id % 10000000 as string), 7,
'0'))"
+ private val microStr =
+ "concat('2019-01-27 11:02:01.', lpad(cast(id % 1000000 as string), 6,
'0'))"
+
+ private def doBenchmark(cardinality: Int, exprs: String*): Unit = {
+ spark.range(cardinality).selectExpr(exprs: _*).noop()
+ }
+
+ /** A wholestage on/off codegen benchmark that just projects `exprs` over
`cardinality` rows. */
+ private def runProject(cardinality: Int, name: String, exprs: String*): Unit
= {
+ codegenBenchmark(name, cardinality) {
+ doBenchmark(cardinality, exprs: _*)
+ }
+ }
+
+ /** A codegen benchmark that builds a single `ts` column then runs `op(ds)`.
*/
+ private def runOnColumn(
+ cardinality: Int, name: String, tsExpr: String)(op:
org.apache.spark.sql.DataFrame => Unit)
+ : Unit = {
+ codegenBenchmark(name, cardinality) {
+ op(spark.range(cardinality).selectExpr(s"$tsExpr as ts"))
+ }
+ }
+
+ override def runBenchmarkSuite(mainArgs: Array[String]): Unit = {
+ withDefaultTimeZone(LA) {
+ withSQLConf(
+ SQLConf.SESSION_LOCAL_TIMEZONE.key -> LA.getId,
+ SQLConf.TIMESTAMP_NANOS_TYPES_ENABLED.key -> "true") {
+
+ runBenchmark("Cast") {
+ runProject(N, "micros -> string", s"cast($microNtz as string)")
+ runProject(N, "nanos(9) -> string", s"cast($nanosNtz as string)")
+ runProject(N, "nanos(7) -> string", s"cast($nanosNtz7 as string)")
+ runProject(N, "micros -> nanos(9)", s"cast($microNtz as
timestamp_ntz(9))")
+ runProject(N, "nanos(9) -> micros", s"cast($nanosNtz as
timestamp_ntz)")
+ runProject(N, "nanos(9) -> nanos(7)", s"cast($nanosNtz as
timestamp_ntz(7))")
+ runProject(N, "micros -> date", s"cast($microNtz as date)")
+ runProject(N, "nanos(9) -> date", s"cast($nanosNtz as date)")
+ }
+
+ runBenchmark("Parse from string") {
+ val n = 1000000
+ runProject(n, "string(6 digits) -> micros", s"cast($microStr as
timestamp_ntz)")
+ runProject(n, "string(9 digits) -> nanos(9)", s"cast($nanosStr as
timestamp_ntz(9))")
+ runProject(n, "string(7 digits) -> nanos(7)", s"cast($nanosStr7 as
timestamp_ntz(7))")
+ }
+
+ runBenchmark("Hashing") {
+ runProject(N, "murmur3 hash(micros)", s"hash($microNtz)")
+ runProject(N, "murmur3 hash(nanos(9))", s"hash($nanosNtz)")
+ runProject(N, "xxhash64(micros)", s"xxhash64($microNtz)")
+ runProject(N, "xxhash64(nanos(9))", s"xxhash64($nanosNtz)")
+ }
+
+ runBenchmark("Sort (global ORDER BY)") {
+ runOnColumn(N, "order by micros", microNtz)(_.orderBy("ts").noop())
+ runOnColumn(N, "order by nanos(9)", nanosNtz)(_.orderBy("ts").noop())
+ }
+
+ runBenchmark("Group-by aggregation (~1M distinct keys)") {
+ val microKey = "timestamp_seconds(id % 1000000)"
+ val nanosKey = "timestamp_nanos((id % 1000000) * 1000000000 + (id %
1000000) % 1000)"
+ runOnColumn(N, "group by micros key",
microKey)(_.groupBy("ts").count().noop())
+ runOnColumn(N, "group by nanos(9) key",
nanosKey)(_.groupBy("ts").count().noop())
+ }
+
+ runBenchmark("MIN/MAX aggregation") {
+ runProject(N, "min/max micros", s"min($microNtz) as mn",
s"max($microNtz) as mx")
+ runProject(N, "min/max nanos(9)", s"min($nanosNtz) as mn",
s"max($nanosNtz) as mx")
+ }
+
+ runBenchmark("Datetime functions") {
+ runProject(N, "hour(micros)", s"hour($microLtz)")
+ runProject(N, "hour(nanos(9))", s"hour($nanosLtz)")
+ runProject(N, "second(micros)", s"second($microLtz)")
+ runProject(N, "second(nanos(9))", s"second($nanosLtz)")
+ runProject(N, "extract(SECOND, micros)", s"extract(SECOND from
$microLtz)")
+ runProject(N, "extract(SECOND, nanos(9))", s"extract(SECOND from
$nanosLtz)")
+ // unix_nanos is nanos-only (requires p in [7,9]); the micro analog
is unix_micros.
+ runProject(N, "unix_micros(micros)", s"unix_micros($microLtz)")
+ runProject(N, "unix_nanos(nanos(9))", s"unix_nanos($nanosLtz)")
+ runProject(N, "micros + interval 1 hour", s"$microLtz + interval 1
hour")
+ runProject(N, "nanos(9) + interval 1 hour", s"$nanosLtz + interval 1
hour")
+ }
+
+ runBenchmark("Parquet write/read") {
+ withTempPath { path =>
+ val microPath = s"${path.getAbsolutePath}/micros"
+ val nanosPath = s"${path.getAbsolutePath}/nanos"
+
+ val write = new Benchmark("Save to Parquet", N, output = output)
+ write.addCase("micros (TIMESTAMP_NTZ)", 1) { _ =>
+ spark.range(N).selectExpr(s"$microNtz as ts")
+ .write.mode("overwrite").format("parquet").save(microPath)
+ }
+ write.addCase("nanos(9) (TIMESTAMP_NTZ(9))", 1) { _ =>
+ spark.range(N).selectExpr(s"$nanosNtz as ts")
+ .write.mode("overwrite").format("parquet").save(nanosPath)
+ }
+ write.run()
+
+ val read = new Benchmark("Load from Parquet", N, output = output)
+ Seq(true, false).foreach { vec =>
+ read.addCase(s"micros, vectorized ${if (vec) "on" else "off"}",
3) { _ =>
+ withSQLConf(SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key ->
vec.toString) {
+ spark.read.format("parquet").load(microPath).noop()
+ }
+ }
+ }
+ // For nanos, vectorized read is disabled internally regardless of
the flag - running
+ // both documents that there is no vectorized fast path yet.
+ Seq(true, false).foreach { vec =>
+ val name =
+ s"nanos(9), vectorized ${if (vec) "on" else "off"} (row-based)"
+ read.addCase(name, 3) { _ =>
+ withSQLConf(SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key ->
vec.toString) {
+ spark.read.format("parquet").load(nanosPath).noop()
+ }
+ }
+ }
+ read.run()
+ }
+ }
+
+ runBenchmark("ORC write/read") {
+ withTempPath { path =>
+ val microPath = s"${path.getAbsolutePath}/micros"
+ val nanosPath = s"${path.getAbsolutePath}/nanos"
+
+ val write = new Benchmark("Save to ORC", N, output = output)
+ write.addCase("micros (TIMESTAMP_NTZ)", 1) { _ =>
+ spark.range(N).selectExpr(s"$microNtz as ts")
+ .write.mode("overwrite").format("orc").save(microPath)
+ }
+ write.addCase("nanos(9) (TIMESTAMP_NTZ(9))", 1) { _ =>
+ spark.range(N).selectExpr(s"$nanosNtz as ts")
+ .write.mode("overwrite").format("orc").save(nanosPath)
+ }
+ write.run()
+
+ val read = new Benchmark("Load from ORC", N, output = output)
+ Seq(true, false).foreach { vec =>
+ read.addCase(s"micros, vectorized ${if (vec) "on" else "off"}",
3) { _ =>
+ withSQLConf(SQLConf.ORC_VECTORIZED_READER_ENABLED.key ->
vec.toString) {
+ spark.read.format("orc").load(microPath).noop()
+ }
+ }
+ }
+ Seq(true, false).foreach { vec =>
+ read.addCase(s"nanos(9), vectorized ${if (vec) "on" else
"off"}", 3) { _ =>
+ withSQLConf(SQLConf.ORC_VECTORIZED_READER_ENABLED.key ->
vec.toString) {
+ spark.read.format("orc").load(nanosPath).noop()
+ }
+ }
+ }
+ read.run()
+ }
+ }
+ }
+ }
+ }
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
