This is an automated email from the ASF dual-hosted git repository. quinnj pushed a commit to branch preservation/storage-20261008/julia-dev-Arrow-decimals-f2071e12 in repository https://gitbox.apache.org/repos/asf/arrow-julia.git
commit aed02685161a1dd866f3f7f2ffe4e3a3c1e513a9 Author: Jacob Quinn <[email protected]> AuthorDate: Tue Sep 1 10:58:21 2026 -0600 Document decimal columns as Decimals.jl values Update the read and write type-mapping tables, the 2.x migration note (the `Arrow.Decimal` type is gone, and 2.x could only write 128/256), the nesting caveat, and the changelog. Co-Authored-By: Claude Fable 5 <[email protected]> --- CHANGELOG.md | 14 +++++++++++++- docs/src/manual.md | 13 +++++++++---- docs/src/migration.md | 16 ++++++++++++---- 3 files changed, 34 insertions(+), 9 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1745beb..e12f1d7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -63,11 +63,23 @@ writer, validation, scan, and C interface engines. Packages that define custom mappings should still depend on and import ArrowTypes.jl directly. - Big-endian IPC and delta-dictionary messages are rejected. +- Decimal columns read and write as Decimals.jl `Decimal{precision,scale,T}` + instead of the `Arrow.Decimal` type Arrow 2.x defined itself. The 2.x type + is gone, and Arrow 2.x could only write 128- and 256-bit decimals. - Arrow 3.0 requires ArrowTypes.jl 2.x, Tables.jl 1.14 (the first release - with `Tables.Scan`), and the ArrowStrings.jl 1.0 release. + with `Tables.Scan`), the ArrowStrings.jl 1.0 release, and Decimals.jl 1.x. ### Added +- Full decimal support over Decimals.jl. All four Arrow widths + (decimal32/64/128/256, the 32- and 64-bit ones being format 1.5 additions) + read as `Decimal{precision,scale,T}` and write back at the bit width `T` + names. The Julia coefficient is byte-identical to an Arrow decimal slot, so + reads take one bulk copy and a non-nullable write is zero-copy — no + per-element conversion in either direction. A `DecimalValue` column, whose + scale travels with each value, is refused with an `ArgumentError` naming + `rescale`; so is a decimal descriptor whose scale falls outside + `0:precision`, which reads in Arrow's raw coefficient domain instead. - `Tables.Scan` projection, filter, limit, and offset pushdown. - Typed `Tables.Scan` select overrides for composite rows: a `:column => NamedTuple{...}` select item reads a Struct column as typed diff --git a/docs/src/manual.md b/docs/src/manual.md index d283497..d04c033 100644 --- a/docs/src/manual.md +++ b/docs/src/manual.md @@ -197,8 +197,7 @@ maps to `Union{Missing, T}`. | Timestamp (microsecond, nanosecond) | `Int64` (raw storage — `DateTime` cannot represent it) | | Time32/Time64 | `Dates.Time` | | Duration | `Dates.Second`/`Millisecond`/`Microsecond`/`Nanosecond` by unit | -| Decimal32/64 | `Int32`/`Int64` (unscaled integer storage) | -| Decimal128/256 | `Vector{UInt8}` (raw native-endian storage) | +| Decimal32/64/128/256 | `Decimals.Decimal{precision,scale,T}` with `T` the width's `Int32`/`Int64`/`Int128`/`Int256` (a descriptor whose scale falls outside `0:precision`, which `Decimal{P,S}` cannot name, keeps the raw storage: `Int32`/`Int64`, or `Vector{UInt8}` at 128/256 bits) | | Interval | `Int32` (year-month) or a `NamedTuple` (day-time, month-day-nano) | | Dictionary-encoded scalar | the mapping of the *value* type (indices are resolved) | @@ -441,13 +440,19 @@ At the *top level* of a column the facade adds: | `Dates.DateTime` | Timestamp (millisecond) | | `Dates.Time` | Time64 (nanosecond) | | `Dates.Second/Millisecond/Microsecond/Nanosecond` | Duration of that unit | +| `Decimals.Decimal{P,S,T}` | Decimal of precision `P`, scale `S`, and the bit width of `T` (32/64/128/256), **zero-copy** when the column declares no `Missing` | | `NamedTuple` whose fields are core columns | Struct; `Union{Missing, T}` adds parent validity while child nullability stays declared | | `Arrow.DictEncode` over a writable column | Dictionary of the recursive mapping of its values | | `ArrowStrings.StringVector` | Utf8View, **zero-copy** — the column's memory is the Arrow array (see below) | These native facade conversions do not recurse through list or struct shapes: -a `Vector{Date}` inside a list, or a `Date` or `SubString` field of a -`NamedTuple`, is refused with an `ArgumentError` naming the element type. A +a `Vector{Date}` or `Vector{Decimal{P,S,T}}` inside a list, or a `Date` or +`SubString` field of a `NamedTuple`, is refused with an `ArgumentError` naming +the element type. Reading a foreign file whose decimals are nested that way +still works — those values arrive in Arrow's raw coefficient domain, as nested +dates arrive as storage integers — and rewriting one preserves its descriptor. +`Decimals.DecimalValue` carries a scale per value, which an Arrow decimal +schema cannot express; `rescale` such a column to a fixed `Decimal{P,S}` first. A top-level `DictEncode` pool uses the same native column mapping as an ordinary top-level column. The ArrowTypes.jl mappings described below do recurse. A column with element type `Any` is narrowed once (recovering list columns of a diff --git a/docs/src/migration.md b/docs/src/migration.md index f2d14fe..8afa28b 100644 --- a/docs/src/migration.md +++ b/docs/src/migration.md @@ -53,10 +53,18 @@ Use `Tables.getcolumn(table, name_or_index)` in place of Arrow 2.x table indexing. Decimal read types changed. Arrow 2.x read decimal columns as `Arrow.Decimal` -values that wrap a BitIntegers.jl `Int128` or `Int256` unscaled integer. Arrow -3.0 reads Decimal32 and Decimal64 as unscaled `Int32` and `Int64` values, and -Decimal128 and Decimal256 as raw native-endian byte vectors. See -[Type mapping when reading](@ref) for the complete table. +values — a type Arrow.jl defined itself — that wrap an `Int128` or `Int256` +unscaled integer, and could only write those two widths. Arrow 3.0 reads and +writes all four widths as +[Decimals.jl](https://github.com/JuliaData/Decimals.jl)'s +`Decimal{precision,scale,T}`, a full-featured decimal number type rather than +a storage wrapper. Its coefficient is byte-identical to an Arrow decimal slot, +so columns cross in both directions with no per-element conversion. Writing a +`Decimal{P,S,T}` column produces the bit width `T` names — Decimal32 and +Decimal64 are format 1.5 additions, so a reader older than that needs the +column widened to 128 bits first. A `DecimalValue`, whose scale travels with +each value, cannot be written: `rescale` it to a fixed `Decimal{P,S}` first. +See [Type mapping when reading](@ref) for the complete table. Struct row types changed. Arrow 2.x read a plain (non-extension) Struct column as `NamedTuple` rows. Arrow 3.0 reads it as `Vector{Pair{String,Any}}`
