Thank you for sharing the direction, Max.
Since this is internal refactoring, can we do this migration safely in a
step-by-step manner over multiple Apache Spark versions without blocking any
Apache Spark releases?
The proposed direction itself looks reasonable and doable for me.
Thanks,
Dongj
I think this is a great idea. There is a signifcant backlog of types which
should be added: E.g TIMESTAMP(9), TIMESTAMP WITH TIME ZONE, TIME WITH
TIMEZONE, some sort of big decimal to name a few).
Making these more "plug and play" is goodness.
+1
On Sep 10, 2025, at 1:22 PM, Max Gekk wrote:
H
Hi All,
I would like to propose refactoring of internal operations over Catalyst's
data types. In the current implementation, data types are handled in an
adhoc manner, and processing logic is dispersed across the entire code
base. There are more than 100 places where every data type is pattern
m
