Nikolay,
If there are multiple configuration in XML, IgniteContext will always use
only one of them. Looks like current approach simply doesn't work. I
propose to report schema name as 'database' in Spark. If there are multiple
clients, you would create multiple sessions and multiple catalogs.
Makes sense?
-Val
On Fri, Aug 24, 2018 at 12:33 AM Nikolay Izhikov <nizhi...@apache.org>
wrote:
> Hello, Valentin.
>
> > catalog exist in scope of a single IgniteSparkSession> (and therefore
> single IgniteContext and single Ignite instance)?
>
> Yes.
> Actually, I was thinking about use case when we have several Ignite
> configuration in one XML file.
> Now I see, may be this is too rare use-case to support.
>
> Stuart, Valentin, What is your proposal?
>
> В Ср, 22/08/2018 в 08:56 -0700, Valentin Kulichenko пишет:
> > Nikolay,
> >
> > Whatever we decide on would be right :) Basically, we need to answer this
> > question: does the catalog exist in scope of a single IgniteSparkSession
> > (and therefore single IgniteContext and single Ignite instance)? In other
> > words, in case of a rare use case when a single Spark application
> connects
> > to multiple Ignite clusters, would there be a catalog created per
> cluster?
> >
> > If the answer is yes, current logic doesn't make sense.
> >
> > -Val
> >
> >
> > On Wed, Aug 22, 2018 at 1:44 AM Nikolay Izhikov <nizhi...@apache.org>
> wrote:
> >
> > > Hello, Valentin.
> > >
> > > > I believe we should get rid of this logic and use Ignite schema name
> as
> > >
> > > database name in Spark's catalog.
> > >
> > > When I develop Ignite integration with Spark Data Frame I use following
> > > abstraction described by Vladimir Ozerov:
> > >
> > > "1) Let's consider Ignite cluster as a single database ("catalog" in
> ANSI
> > > SQL'92 terms)." [1]
> > >
> > > Am I was wrong? If yes - let's fix it.
> > >
> > > [1]
> > >
> http://apache-ignite-developers.2346864.n4.nabble.com/SQL-usability-catalogs-schemas-and-tables-td17148.html
> > >
> > > В Ср, 22/08/2018 в 09:26 +0100, Stuart Macdonald пишет:
> > > > Hi Val, yes that's correct. I'd be happy to make the change to have
> the
> > > > database reference the schema if Nikolay agrees. (I'll first need to
> do a
> > > > bit of research into how to obtain the list of all available
> schemata...)
> > > >
> > > > Thanks,
> > > > Stuart.
> > > >
> > > > On Tue, Aug 21, 2018 at 9:43 PM, Valentin Kulichenko <
> > > > valentin.kuliche...@gmail.com> wrote:
> > > >
> > > > > Stuart,
> > > > >
> > > > > Thanks for pointing this out, I was not aware that we use Spark
> > >
> > > database
> > > > > concept this way. Actually, this confuses me a lot. As far as I
> > >
> > > understand,
> > > > > catalog is created in the scope of a particular IgniteSparkSession,
> > >
> > > which
> > > > > in turn is assigned to a particular IgniteContext and therefore
> single
> > > > > Ignite client. If that's the case, I don't think it should be
> aware of
> > > > > other Ignite clients that are connected to other clusters. This
> doesn't
> > > > > look like correct behavior to me, not to mention that with this
> > >
> > > approach
> > > > > having multiple databases would be a very rare case. I believe we
> > >
> > > should
> > > > > get rid of this logic and use Ignite schema name as database name
> in
> > > > > Spark's catalog.
> > > > >
> > > > > Nikolay, what do you think?
> > > > >
> > > > > -Val
> > > > >
> > > > > On Tue, Aug 21, 2018 at 8:17 AM Stuart Macdonald <
> stu...@stuwee.org>
> > > > > wrote:
> > > > >
> > > > > > Nikolay, Val,
> > > > > >
> > > > > > The JDBC Spark datasource[1] -- as far as I can tell -- has no
> > > > > > ExternalCatalog implementation, it just uses the database
> specified
> > >
> > > in the
> > > > > > JDBC URL. So I don't believe there is any way to call
> listTables() or
> > > > > > listDatabases() for JDBC provider.
> > > > > >
> > > > > > The Hive ExternalCatalog[2] makes the distinction between
> database
> > >
> > > and
> > > > > > table using the actual database and table mechanisms built into
> the
> > > > > > catalog, which is fine because Hive has the clear distinction and
> > > > > > hierarchy
> > > > > > of databases and tables.
> > > > > >
> > > > > > *However* Ignite already uses the "database" concept in the
> Ignite
> > > > > >
> > > > > > ExternalCatalog[3] to mean the name of an Ignite instance. So in
> > >
> > > Ignite we
> > > > > > have instances containing schemas containing tables, and Spark
> only
> > >
> > > has
> > > > > > the
> > > > > > concept of databases and tables so it seems like either we ignore
> > >
> > > one of
> > > > > > the three Ignite concepts or combine two of them into database or
> > >
> > > table.
> > > > > > The current implementation in the pull request combines Ignite
> > >
> > > schema and
> > > > > > table attributes into the Spark table attribute.
> > > > > >
> > > > > > Stuart.
> > > > > >
> > > > > > [1]
> > > > > > https://github.com/apache/spark/blob/master/sql/core/
> > > > > > src/main/scala/org/apache/spark/sql/execution/
> > > > > > datasources/jdbc/JDBCRelation.scala
> > > > > > [2]
> > > > > > https://github.com/apache/spark/blob/master/sql/hive/
> > > > > >
> src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala
> > > > > > [3]
> > > > > > https://github.com/apache/ignite/blob/master/modules/
> > > > > > spark/src/main/scala/org/apache/spark/sql/ignite/
> > > > > > IgniteExternalCatalog.scala
> > > > > >
> > > > > > On Tue, Aug 21, 2018 at 9:31 AM, Nikolay Izhikov <
> > >
> > > nizhi...@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > > Hello, Stuart.
> > > > > > >
> > > > > > > Can you do some research and find out how schema is handled in
> Data
> > > > > >
> > > > > > Frames
> > > > > > > for a regular RDBMS such as Oracle, MySQL, etc?
> > > > > > >
> > > > > > > В Пн, 20/08/2018 в 15:37 -0700, Valentin Kulichenko пишет:
> > > > > > > > Stuart, Nikolay,
> > > > > > > >
> > > > > > > > I see that the 'Table' class (returned by listTables method)
> has
> > >
> > > a
> > > > > > >
> > > > > > > 'database' field. Can we use this one to report schema name?
> > > > > > > >
> > > > > > > > In any case, I think we should look into how this is done in
> data
> > > > > >
> > > > > > source
> > > > > > > implementations for other databases. Any relational database
> has a
> > > > > >
> > > > > > notion
> > > > > > > of schema, and I'm sure Spark integrations take this into
> account
> > > > > >
> > > > > > somehow.
> > > > > > > >
> > > > > > > > -Val
> > > > > > > >
> > > > > > > > On Mon, Aug 20, 2018 at 6:12 AM Nikolay Izhikov <
> > >
> > > nizhi...@apache.org>
> > > > > > >
> > > > > > > wrote:
> > > > > > > > > Hello, Stuart.
> > > > > > > > >
> > > > > > > > > Personally, I think we should change current tables naming
> and
> > > > > >
> > > > > > return
> > > > > > > table in form of `schema.table`.
> > > > > > > > >
> > > > > > > > > Valentin, could you share your opinion?
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > В Пн, 20/08/2018 в 10:04 +0100, Stuart Macdonald пишет:
> > > > > > > > > > Igniters,
> > > > > > > > > >
> > > > > > > > > > While reviewing the changes for IGNITE-9228 [1,2],
> Nikolay
> > >
> > > and I
> > > > > >
> > > > > > are
> > > > > > > > > > discussing whether to introduce a change which may impact
> > > > > >
> > > > > > backwards
> > > > > > > > > > compatibility; Nikolay suggested we take the discussion
> to
> > >
> > > this
> > > > > >
> > > > > > list.
> > > > > > > > > >
> > > > > > > > > > Ignite implements a custom Spark catalog which provides
> an
> > >
> > > API by
> > > > > > >
> > > > > > > which
> > > > > > > > > > Spark users can list the tables which are available in
> Ignite
> > > > > >
> > > > > > which
> > > > > > > can be
> > > > > > > > > > queried via Spark SQL. Currently that table name list
> > >
> > > includes
> > > > > >
> > > > > > just
> > > > > > > the
> > > > > > > > > > names of the tables, but IGNITE-9228 is introducing a
> change
> > >
> > > which
> > > > > > >
> > > > > > > allows
> > > > > > > > > > optional prefixing of schema names to table names to
> > >
> > > disambiguate
> > > > > > >
> > > > > > > multiple
> > > > > > > > > > tables with the same name in different schemas. For the
> "list
> > > > > > >
> > > > > > > tables" API
> > > > > > > > > > we therefore have two options:
> > > > > > > > > >
> > > > > > > > > > 1. List the tables using both their table names and
> > > > > >
> > > > > > schema-qualified
> > > > > > > table
> > > > > > > > > > names (eg. [ "myTable", "mySchema.myTable" ]) even though
> > >
> > > they are
> > > > > > >
> > > > > > > the same
> > > > > > > > > > underlying table. This retains backwards compatibility
> with
> > >
> > > users
> > > > > >
> > > > > > who
> > > > > > > > > > expect "myTable" to appear in the catalog.
> > > > > > > > > > 2. List the tables using only their schema-qualified
> names.
> > >
> > > This
> > > > > > >
> > > > > > > eliminates
> > > > > > > > > > duplication of names in the catalog but will potentially
> > >
> > > break
> > > > > > > > > > compatibility with users who expect the table name in the
> > >
> > > catalog.
> > > > > > > > > >
> > > > > > > > > > With either option we will allow for Spark SQL SELECT
> > >
> > > statements
> > > > > >
> > > > > > to
> > > > > > > use
> > > > > > > > > > either table name or schema-qualified table names, this
> > >
> > > change
> > > > > >
> > > > > > would
> > > > > > > purely
> > > > > > > > > > impact the API which is used to list available tables.
> > > > > > > > > >
> > > > > > > > > > Any opinions would be welcome.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Stuart.
> > > > > > > > > >
> > > > > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-9228
> > > > > > > > > > [2] https://github.com/apache/ignite/pull/4551
