> If you just want to save typing the catalog name when writing table
names, you can set your custom catalog as the default catalog (See
SQLConf.DEFAULT_CATALOG). SQLConf.V2_SESSION_CATALOG_IMPLEMENTATION is used
to extend the v1 session catalog, not replace it.
I'm sorry, but I don't get this.
I disagree that this is “by design”. An operation like DROP TABLE should
use a v2 drop plan if the table is v2.
If a v2 table is loaded or created using a v2 catalog it should also be
dropped that way. Otherwise, the v2 catalog is not notified when the table
is dropped and can’t perform other
If you just want to save typing the catalog name when writing table names,
you can set your custom catalog as the default catalog (See
SQLConf.DEFAULT_CATALOG). SQLConf.V2_SESSION_CATALOG_IMPLEMENTATION is used
to extend the v1 session catalog, not replace it.
On Wed, Oct 7, 2020 at 5:36 PM
If it's by design and not prepared, then IMHO replacing the default session
catalog is better to be restricted until things are sorted out, as it gives
pretty much confusion and has known bugs. Actually there's another
bug/limitation on default session catalog on the length of identifier,
so
Ah, this is by design. V1 tables should still go through the v1 session
catalog. I think we can remove this restriction when we are confident about
the new v2 DDL commands that work with v2 catalog APIs.
On Wed, Oct 7, 2020 at 5:00 PM Jungtaek Lim
wrote:
> My case is DROP TABLE and DROP TABLE
My case is DROP TABLE and DROP TABLE supports both v1 and v2 (as it simply
works when I use custom catalog without replacing the default catalog).
It just fails on v2 when the "default catalog" is replaced (say I replace
'spark_catalog'), because TempViewOrV1Table is providing value even with v2
Not all the DDL commands support v2 catalog APIs (e.g. CREATE TABLE LIKE),
so it's possible that some commands still go through the v1 session catalog
although you configured a custom v2 session catalog.
Can you create JIRA tickets if you hit any DDL commands that don't support
v2 catalog? We
The logical plan for the parsed statement is getting converted either for
old one or v2, and for the former one it keeps using an external catalog
(Hive) - so replacing default session catalog with custom one and trying to
use it like it is in external catalog doesn't work, which destroys the
I've hit this with `DROP TABLE` commands that should be passed to a
registered v2 session catalog, but are handled by v1. I think that's the
only case we hit in our downstream test suites, but we haven't been
exploring the use of a session catalog for fallback. We use v2 for
everything now, which
Hi devs,
I'm not sure whether it's addressed in Spark 3.1, but at least from Spark
3.0.1, many SQL DDL statements don't seem to go through the custom catalog
when I replace default catalog with custom catalog and only provide
'dbName.tableName' as table identifier.
I'm not an expert in this
10 matches
Mail list logo
