It is important to consider potential impacts on Spark tables stored
in the Hive metastore during an "upgrade". Depending on the upgrade
path, the Hive metastore schema or SerDes behavior might change,
requiring adjustments in the Sparkark code
or configurations. I mentioned the need to test the
Yes, Spark has a shim layer to support all Hive versions. It shouldn't be
an issue as many users create native Spark data source tables already
today, by explicitly putting the `USING` clause in the CREATE TABLE
statement.
On Wed, May 1, 2024 at 12:56 AM Mich Talebzadeh
wrote:
> @Wenchen Fan
@Wenchen Fan Got your explanation, thanks!
My understanding is that even if we create Spark tables using Spark's
native data sources, by default, the metadata about these tables will
be stored in the Hive metastore. As a consequence, a Hive upgrade can
potentially affect Spark tables. For
