This is an automated email from the ASF dual-hosted git repository.
voonhous pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 2f77af6096af docs(sql-ddl): partition columns must be declared last in
CREATE TABLE (#19555)
2f77af6096af is described below
commit 2f77af6096af3c3a918092edf34b620d72f0adbb
Author: deepakpanda93 <[email protected]>
AuthorDate: Sun Aug 9 12:46:51 2026 +0530
docs(sql-ddl): partition columns must be declared last in CREATE TABLE
(#19555)
Spark moves partition columns to the end of the table schema. Declaring one
earlier makes the stored column order differ from what was written, and a
positional INSERT then assigns values to the wrong columns.
Verified on Spark 3.5.7 with Hudi 1.2.0. Declaring
(id, name, price, dt, ts) with PARTITIONED BY (dt) stores the table as
(id, name, price, ts, dt), and `insert into t select
1,'a1',10.0,'2021-03-21',1L`
fails with
[INCOMPATIBLE_DATA_FOR_TABLE.CANNOT_SAFELY_CAST]
Cannot safely cast `ts` "STRING" to "BIGINT"
which is the symptom reported in the issue. Worse, when the shifted columns
happen to be type-compatible there is no error at all: declaring
(id, a, dt, b) as strings and inserting 1,'VALUE_A','VALUE_DT','VALUE_B'
returns exit 0 and stores b='VALUE_DT' with dt='VALUE_B', putting the
partition value in a data column and vice versa.
The example in this section was an instance of that. With schema
(id, name, dt, hh) and PARTITIONED BY (dt), hh trails the partition column,
so Spark stores (id, name, hh, dt) and a reader following the example
silently gets hh='2024-01-01' and dt='10'. Partitioning by both columns
fixes it without touching the schema: dt and hh are then the trailing
columns already, in declaration order, so nothing is reordered and the
positional insert lands correctly. Measured on all four variants of this
schema -- PARTITIONED BY (dt) misplaces values, (dt, hh) and (hh) are
correct, and (hh, dt) is rejected outright by
HoodieSchemaUtils.checkPartitionSchemaOrder.
The multi-field note is unchanged and remains correct.
Applied to next and to every 1.x versioned copy, which all carried the same
broken example. This is a correction rather than an addition, so it follows
the wider backport used in apache/hudi#19459 rather than the
next-plus-current convention. Re-ran the decisive pair against the 1.0.2
bundle to confirm the older releases behave identically: PARTITIONED BY (dt)
stores (id, name, hh, dt) and yields dt='10', hh='2024-01-01', while
(dt, hh) leaves the order untouched and lands correctly.
The 0.14.x and 0.15.x copies carry the same broken example and are left
alone as end-of-life.
Closes #17357.
Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
---
website/docs/sql_ddl.md | 14 ++++++++++++--
website/versioned_docs/version-1.0.0/sql_ddl.md | 12 +++++++++++-
website/versioned_docs/version-1.0.1/sql_ddl.md | 12 +++++++++++-
website/versioned_docs/version-1.0.2/sql_ddl.md | 12 +++++++++++-
website/versioned_docs/version-1.1.1/sql_ddl.md | 12 +++++++++++-
website/versioned_docs/version-1.2.0/sql_ddl.md | 14 ++++++++++++--
6 files changed, 68 insertions(+), 8 deletions(-)
diff --git a/website/docs/sql_ddl.md b/website/docs/sql_ddl.md
index f18f14812686..150a8197ffc0 100644
--- a/website/docs/sql_ddl.md
+++ b/website/docs/sql_ddl.md
@@ -2,7 +2,7 @@
title: SQL DDL
summary: "In this page, we discuss using SQL DDL commands with Hudi"
toc: true
-last_modified_at: 2026-05-29T00:00:00-00:00
+last_modified_at: 2026-08-07T20:54:32+05:30
---
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
@@ -63,7 +63,7 @@ CREATE TABLE IF NOT EXISTS hudi_table_partitioned (
TBLPROPERTIES (
type = 'cow'
)
-PARTITIONED BY (dt);
+PARTITIONED BY (dt, hh);
```
:::note
@@ -73,6 +73,16 @@ in the same order as they appear in the `CREATE TABLE`
schema. For example, for
should be specified as `PARTITIONED BY (dt, hh)`.
:::
+:::caution
+Declare partition columns last in the `CREATE TABLE` column list. Spark moves
partition columns to the end of the table
+schema, so declaring one earlier makes the stored column order differ from
what you wrote. Declaring
+`(id, name, price, dt, ts)` with `PARTITIONED BY (dt)`, for example, stores
the table as `(id, name, price, ts, dt)`.
+A positional `INSERT INTO ... SELECT` then assigns values to the wrong
columns: it fails with `CANNOT_SAFELY_CAST` when
+the shifted types are incompatible, and writes values to the wrong columns
without reporting an error when they are
+compatible. Naming the columns explicitly, as in `INSERT INTO tbl (id, name,
price, dt, ts) SELECT ...`, also avoids the
+mismatch.
+:::
+
### Create table with record keys and ordering fields
As discussed [here](quick-start-guide.md#keys), tables track each record in
the table using a record key. Hudi auto-generated a highly compressed
diff --git a/website/versioned_docs/version-1.0.0/sql_ddl.md
b/website/versioned_docs/version-1.0.0/sql_ddl.md
index 565a62f480ef..edb256b96b32 100644
--- a/website/versioned_docs/version-1.0.0/sql_ddl.md
+++ b/website/versioned_docs/version-1.0.0/sql_ddl.md
@@ -63,7 +63,7 @@ CREATE TABLE IF NOT EXISTS hudi_table_partitioned (
TBLPROPERTIES (
type = 'cow'
)
-PARTITIONED BY (dt);
+PARTITIONED BY (dt, hh);
```
:::note
@@ -73,6 +73,16 @@ in the same order as they appear in the `CREATE TABLE`
schema. For example, for
should be specified as `PARTITIONED BY (dt, hh)`.
:::
+:::caution
+Declare partition columns last in the `CREATE TABLE` column list. Spark moves
partition columns to the end of the table
+schema, so declaring one earlier makes the stored column order differ from
what you wrote. Declaring
+`(id, name, price, dt, ts)` with `PARTITIONED BY (dt)`, for example, stores
the table as `(id, name, price, ts, dt)`.
+A positional `INSERT INTO ... SELECT` then assigns values to the wrong
columns: it fails with `CANNOT_SAFELY_CAST` when
+the shifted types are incompatible, and writes values to the wrong columns
without reporting an error when they are
+compatible. Naming the columns explicitly, as in `INSERT INTO tbl (id, name,
price, dt, ts) SELECT ...`, also avoids the
+mismatch.
+:::
+
### Create table with record keys and ordering fields
As discussed [here](quick-start-guide.md#keys), tables track each record in
the table using a record key. Hudi auto-generated a highly compressed
diff --git a/website/versioned_docs/version-1.0.1/sql_ddl.md
b/website/versioned_docs/version-1.0.1/sql_ddl.md
index 565a62f480ef..edb256b96b32 100644
--- a/website/versioned_docs/version-1.0.1/sql_ddl.md
+++ b/website/versioned_docs/version-1.0.1/sql_ddl.md
@@ -63,7 +63,7 @@ CREATE TABLE IF NOT EXISTS hudi_table_partitioned (
TBLPROPERTIES (
type = 'cow'
)
-PARTITIONED BY (dt);
+PARTITIONED BY (dt, hh);
```
:::note
@@ -73,6 +73,16 @@ in the same order as they appear in the `CREATE TABLE`
schema. For example, for
should be specified as `PARTITIONED BY (dt, hh)`.
:::
+:::caution
+Declare partition columns last in the `CREATE TABLE` column list. Spark moves
partition columns to the end of the table
+schema, so declaring one earlier makes the stored column order differ from
what you wrote. Declaring
+`(id, name, price, dt, ts)` with `PARTITIONED BY (dt)`, for example, stores
the table as `(id, name, price, ts, dt)`.
+A positional `INSERT INTO ... SELECT` then assigns values to the wrong
columns: it fails with `CANNOT_SAFELY_CAST` when
+the shifted types are incompatible, and writes values to the wrong columns
without reporting an error when they are
+compatible. Naming the columns explicitly, as in `INSERT INTO tbl (id, name,
price, dt, ts) SELECT ...`, also avoids the
+mismatch.
+:::
+
### Create table with record keys and ordering fields
As discussed [here](quick-start-guide.md#keys), tables track each record in
the table using a record key. Hudi auto-generated a highly compressed
diff --git a/website/versioned_docs/version-1.0.2/sql_ddl.md
b/website/versioned_docs/version-1.0.2/sql_ddl.md
index 565a62f480ef..edb256b96b32 100644
--- a/website/versioned_docs/version-1.0.2/sql_ddl.md
+++ b/website/versioned_docs/version-1.0.2/sql_ddl.md
@@ -63,7 +63,7 @@ CREATE TABLE IF NOT EXISTS hudi_table_partitioned (
TBLPROPERTIES (
type = 'cow'
)
-PARTITIONED BY (dt);
+PARTITIONED BY (dt, hh);
```
:::note
@@ -73,6 +73,16 @@ in the same order as they appear in the `CREATE TABLE`
schema. For example, for
should be specified as `PARTITIONED BY (dt, hh)`.
:::
+:::caution
+Declare partition columns last in the `CREATE TABLE` column list. Spark moves
partition columns to the end of the table
+schema, so declaring one earlier makes the stored column order differ from
what you wrote. Declaring
+`(id, name, price, dt, ts)` with `PARTITIONED BY (dt)`, for example, stores
the table as `(id, name, price, ts, dt)`.
+A positional `INSERT INTO ... SELECT` then assigns values to the wrong
columns: it fails with `CANNOT_SAFELY_CAST` when
+the shifted types are incompatible, and writes values to the wrong columns
without reporting an error when they are
+compatible. Naming the columns explicitly, as in `INSERT INTO tbl (id, name,
price, dt, ts) SELECT ...`, also avoids the
+mismatch.
+:::
+
### Create table with record keys and ordering fields
As discussed [here](quick-start-guide.md#keys), tables track each record in
the table using a record key. Hudi auto-generated a highly compressed
diff --git a/website/versioned_docs/version-1.1.1/sql_ddl.md
b/website/versioned_docs/version-1.1.1/sql_ddl.md
index d1c5ba865bdb..a522fa83515a 100644
--- a/website/versioned_docs/version-1.1.1/sql_ddl.md
+++ b/website/versioned_docs/version-1.1.1/sql_ddl.md
@@ -63,7 +63,7 @@ CREATE TABLE IF NOT EXISTS hudi_table_partitioned (
TBLPROPERTIES (
type = 'cow'
)
-PARTITIONED BY (dt);
+PARTITIONED BY (dt, hh);
```
:::note
@@ -73,6 +73,16 @@ in the same order as they appear in the `CREATE TABLE`
schema. For example, for
should be specified as `PARTITIONED BY (dt, hh)`.
:::
+:::caution
+Declare partition columns last in the `CREATE TABLE` column list. Spark moves
partition columns to the end of the table
+schema, so declaring one earlier makes the stored column order differ from
what you wrote. Declaring
+`(id, name, price, dt, ts)` with `PARTITIONED BY (dt)`, for example, stores
the table as `(id, name, price, ts, dt)`.
+A positional `INSERT INTO ... SELECT` then assigns values to the wrong
columns: it fails with `CANNOT_SAFELY_CAST` when
+the shifted types are incompatible, and writes values to the wrong columns
without reporting an error when they are
+compatible. Naming the columns explicitly, as in `INSERT INTO tbl (id, name,
price, dt, ts) SELECT ...`, also avoids the
+mismatch.
+:::
+
### Create table with record keys and ordering fields
As discussed [here](quick-start-guide.md#keys), tables track each record in
the table using a record key. Hudi auto-generated a highly compressed
diff --git a/website/versioned_docs/version-1.2.0/sql_ddl.md
b/website/versioned_docs/version-1.2.0/sql_ddl.md
index f18f14812686..150a8197ffc0 100644
--- a/website/versioned_docs/version-1.2.0/sql_ddl.md
+++ b/website/versioned_docs/version-1.2.0/sql_ddl.md
@@ -2,7 +2,7 @@
title: SQL DDL
summary: "In this page, we discuss using SQL DDL commands with Hudi"
toc: true
-last_modified_at: 2026-05-29T00:00:00-00:00
+last_modified_at: 2026-08-07T20:54:32+05:30
---
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
@@ -63,7 +63,7 @@ CREATE TABLE IF NOT EXISTS hudi_table_partitioned (
TBLPROPERTIES (
type = 'cow'
)
-PARTITIONED BY (dt);
+PARTITIONED BY (dt, hh);
```
:::note
@@ -73,6 +73,16 @@ in the same order as they appear in the `CREATE TABLE`
schema. For example, for
should be specified as `PARTITIONED BY (dt, hh)`.
:::
+:::caution
+Declare partition columns last in the `CREATE TABLE` column list. Spark moves
partition columns to the end of the table
+schema, so declaring one earlier makes the stored column order differ from
what you wrote. Declaring
+`(id, name, price, dt, ts)` with `PARTITIONED BY (dt)`, for example, stores
the table as `(id, name, price, ts, dt)`.
+A positional `INSERT INTO ... SELECT` then assigns values to the wrong
columns: it fails with `CANNOT_SAFELY_CAST` when
+the shifted types are incompatible, and writes values to the wrong columns
without reporting an error when they are
+compatible. Naming the columns explicitly, as in `INSERT INTO tbl (id, name,
price, dt, ts) SELECT ...`, also avoids the
+mismatch.
+:::
+
### Create table with record keys and ordering fields
As discussed [here](quick-start-guide.md#keys), tables track each record in
the table using a record key. Hudi auto-generated a highly compressed
