[
https://issues.apache.org/jira/browse/SPARK-33403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18070895#comment-18070895
]
L. C. Hsieh commented on SPARK-33403:
-------------------------------------
I investigated this issue and believe the current behavior is correct for the
default case, though there is a subtle edge case worth noting.
In DSv2, CatalogPlugin.defaultNamespace() returns an empty array by default,
meaning the root namespace is the default namespace for most V2 catalogs. When
you run CREATE TABLE testcat.table ..., Spark's name resolution parses this as
catalog=testcat, namespace=[], name=table — placing the table at the root
namespace. Since the root namespace is the default namespace, this is
consistent with how CREATE TABLE table (a single name part) resolves using
catalogManager.currentNamespace, which falls back to defaultNamespace().
So SHOW TABLES displaying an empty string for the namespace column is correct —
it faithfully reflects that the table's Identifier.namespace() is an empty
array, which corresponds to the catalog's default (root) namespace.
The apparent discrepancy with DSv1 comes from the fact that Hive's default
database is a named, explicit namespace, whereas DSv2's default namespace is
the root level (empty array). They serve the same purpose — the implicit
namespace when none is specified — but are represented differently. Displaying
default for a V2 table in the root namespace would be misleading, as there may
not be a namespace called default in that catalog.
However, if a V2 catalog overrides defaultNamespace() to return something other
than an empty array (e.g., ["default"]), then CREATE TABLE testcat.table and
CREATE TABLE table (with testcat as the current catalog) would resolve to
different namespaces — the former to the root namespace and the latter to
default. Whether this inconsistency is intentional or a bug in name resolution
is a separate question from the SHOW TABLES display issue reported here.
I'd suggest resolving this as "Not A Problem".
> DSv2 SHOW TABLES doesn't show `default`
> ---------------------------------------
>
> Key: SPARK-33403
> URL: https://issues.apache.org/jira/browse/SPARK-33403
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 3.1.0
> Reporter: Max Gekk
> Priority: Major
>
> DSv1:
> {code:scala}
> test("namespace is not specified and the default catalog is set") {
> withSQLConf(SQLConf.DEFAULT_CATALOG.key -> catalog) {
> withTable("table") {
> spark.sql(s"CREATE TABLE table (id bigint, data string)
> $defaultUsing")
> sql("SHOW TABLES").show()
> }
> }
> }
> {code}
> {code}
> +--------+---------+-----------+
> |database|tableName|isTemporary|
> +--------+---------+-----------+
> | default| table| false|
> +--------+---------+-----------+
> {code}
> DSv2:
> {code}
> +---------+---------+
> |namespace|tableName|
> +---------+---------+
> | | table|
> +---------+---------+
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
