Nikolay, Val, The JDBC Spark datasource[1] -- as far as I can tell -- has no ExternalCatalog implementation, it just uses the database specified in the JDBC URL. So I don't believe there is any way to call listTables() or listDatabases() for JDBC provider.
The Hive ExternalCatalog[2] makes the distinction between database and table using the actual database and table mechanisms built into the catalog, which is fine because Hive has the clear distinction and hierarchy of databases and tables. *However* Ignite already uses the "database" concept in the Ignite ExternalCatalog[3] to mean the name of an Ignite instance. So in Ignite we have instances containing schemas containing tables, and Spark only has the concept of databases and tables so it seems like either we ignore one of the three Ignite concepts or combine two of them into database or table. The current implementation in the pull request combines Ignite schema and table attributes into the Spark table attribute. Stuart. [1] https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRelation.scala [2] https://github.com/apache/spark/blob/master/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala [3] https://github.com/apache/ignite/blob/master/modules/spark/src/main/scala/org/apache/spark/sql/ignite/IgniteExternalCatalog.scala On Tue, Aug 21, 2018 at 9:31 AM, Nikolay Izhikov <nizhi...@apache.org> wrote: > Hello, Stuart. > > Can you do some research and find out how schema is handled in Data Frames > for a regular RDBMS such as Oracle, MySQL, etc? > > В Пн, 20/08/2018 в 15:37 -0700, Valentin Kulichenko пишет: > > Stuart, Nikolay, > > > > I see that the 'Table' class (returned by listTables method) has a > 'database' field. Can we use this one to report schema name? > > > > In any case, I think we should look into how this is done in data source > implementations for other databases. Any relational database has a notion > of schema, and I'm sure Spark integrations take this into account somehow. > > > > -Val > > > > On Mon, Aug 20, 2018 at 6:12 AM Nikolay Izhikov <nizhi...@apache.org> > wrote: > > > Hello, Stuart. > > > > > > Personally, I think we should change current tables naming and return > table in form of `schema.table`. > > > > > > Valentin, could you share your opinion? > > > > > > > > > В Пн, 20/08/2018 в 10:04 +0100, Stuart Macdonald пишет: > > > > Igniters, > > > > > > > > While reviewing the changes for IGNITE-9228 [1,2], Nikolay and I are > > > > discussing whether to introduce a change which may impact backwards > > > > compatibility; Nikolay suggested we take the discussion to this list. > > > > > > > > Ignite implements a custom Spark catalog which provides an API by > which > > > > Spark users can list the tables which are available in Ignite which > can be > > > > queried via Spark SQL. Currently that table name list includes just > the > > > > names of the tables, but IGNITE-9228 is introducing a change which > allows > > > > optional prefixing of schema names to table names to disambiguate > multiple > > > > tables with the same name in different schemas. For the "list > tables" API > > > > we therefore have two options: > > > > > > > > 1. List the tables using both their table names and schema-qualified > table > > > > names (eg. [ "myTable", "mySchema.myTable" ]) even though they are > the same > > > > underlying table. This retains backwards compatibility with users who > > > > expect "myTable" to appear in the catalog. > > > > 2. List the tables using only their schema-qualified names. This > eliminates > > > > duplication of names in the catalog but will potentially break > > > > compatibility with users who expect the table name in the catalog. > > > > > > > > With either option we will allow for Spark SQL SELECT statements to > use > > > > either table name or schema-qualified table names, this change would > purely > > > > impact the API which is used to list available tables. > > > > > > > > Any opinions would be welcome. > > > > > > > > Thanks, > > > > Stuart. > > > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-9228 > > > > [2] https://github.com/apache/ignite/pull/4551 >
