mattfaltyn opened a new issue, #1498:
URL: https://github.com/apache/iceberg-go/issues/1498
### Apache Iceberg version
main (development)
### Please describe the bug 🐞
## Summary
`BucketTransform` hashes `timestamp_ns` and `timestamptz_ns` values using
their raw nanosecond representation.
The Iceberg specification requires nanosecond timestamps to be converted to
microseconds before hashing. This ensures equivalent microsecond and nanosecond
timestamps produce the same bucket.
## Reproduction
```go
package main
import (
"fmt"
"time"
iceberg "github.com/apache/iceberg-go"
)
func apply(transform iceberg.BucketTransform, value iceberg.Literal)
iceberg.Literal {
return transform.Apply(iceberg.Optional[iceberg.Literal]{
Valid: true,
Val: value,
}).Val
}
func main() {
transform := iceberg.BucketTransform{NumBuckets: 16}
value := time.Date(2017, 11, 16, 22, 31, 8, 1_001, time.UTC)
micros := iceberg.Timestamp(value.UnixMicro())
nanos := iceberg.TimestampNano(value.UnixNano())
fmt.Printf("timestamp: %s\n", apply(transform,
iceberg.NewLiteral(micros)))
fmt.Printf("timestamp_ns: %s\n", apply(transform,
iceberg.NewLiteral(nanos)))
}
```
Run from the repository root:
```shell
go run /tmp/iceberg_timestamp_ns_bucket_repro.go
```
## Actual behavior
```text
timestamp: 6
timestamp_ns: 13
```
The same instant is assigned to different buckets. The mismatch also affects
equality-predicate projection.
## Expected behavior
Both representations should produce bucket `6`.
Per the Iceberg specification, `timestamp_ns` and `timestamptz_ns` must be
converted to microsecond precision before applying the Murmur3 hash.
## Impact
This can create incompatible partition values between iceberg-go and other
Iceberg implementations:
- Go can prune conforming files that contain matching rows.
- Other implementations can prune files written by Go.
- Go-written `bucket[N]` partitions on nanosecond timestamps may be
incompatible with Java Iceberg and PyIceberg.
This can result in silent missing query results rather than a visible error.
## Likely cause
`BucketTransform.Apply` and `BucketTransform.Transformer` currently pass raw
nanoseconds to `hashHelperInt`.
The nanosecond value should first be converted to microseconds using floor
division, including for pre-epoch timestamps.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
