[GitHub] spark pull request: [SPARK-4943][SQL] Allow table name having dot ...
Github user scwf commented on a diff in the pull request:
https://github.com/apache/spark/pull/4062#discussion_r23112796
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala ---
@@ -95,12 +95,12 @@ class HiveContext(sc: SparkContext) extends
SQLContext(sc) {
*/
def refreshTable(tableName: String): Unit = {
// TODO: Database support...
-catalog.refreshTable("default", tableName)
+catalog.refreshTable(Seq("default", tableName))
}
protected[hive] def invalidateTable(tableName: String): Unit = {
// TODO: Database support...
-catalog.invalidateTable("default", tableName)
+catalog.invalidateTable(Seq("default", tableName))
--- End diff --
yes, this just inherited from the old version. Now ```invalidateTable``` do
not support database as you suggest, so here is a todo in line 102, how about
make another PR to fix this? or do it in this PR
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4943][SQL] Allow table name having dot ...
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/4062#discussion_r23111872
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala ---
@@ -95,12 +95,12 @@ class HiveContext(sc: SparkContext) extends
SQLContext(sc) {
*/
def refreshTable(tableName: String): Unit = {
// TODO: Database support...
-catalog.refreshTable("default", tableName)
+catalog.refreshTable(Seq("default", tableName))
}
protected[hive] def invalidateTable(tableName: String): Unit = {
// TODO: Database support...
-catalog.invalidateTable("default", tableName)
+catalog.invalidateTable(Seq("default", tableName))
--- End diff --
Why hardcode default here? When a database isn't specified its up to the
catalog to use the current one, where as this will ignore any `USE` statements
that have been run.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4943][SQL] Allow table name having dot ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4062#issuecomment-70102567 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25602/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4943][SQL] Allow table name having dot ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4062#issuecomment-70102560 [Test build #25602 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25602/consoleFull) for PR 4062 at commit [`e0d1960`](https://github.com/apache/spark/commit/e0d19600204c3a54ca9a6a959ccaaa1c0d7bcdca). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4943][SQL] Allow table name having dot ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4062#issuecomment-70091740 [Test build #25602 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25602/consoleFull) for PR 4062 at commit [`e0d1960`](https://github.com/apache/spark/commit/e0d19600204c3a54ca9a6a959ccaaa1c0d7bcdca). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4943][SQL] Allow table name having dot ...
GitHub user scwf opened a pull request: https://github.com/apache/spark/pull/4062 [SPARK-4943][SQL] Allow table name having dot for db/catalog Follow up for #3941, in this PR 1 enable ddl table name having dot 2 enable hivecontext related api support dot table name with ```tableIdentifier``` You can merge this pull request into a Git repository by running: $ git pull https://github.com/scwf/spark createtable Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/4062.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4062 commit fcab9b047edc8fdc7e088aa83f20375d307dcef3 Author: scwf Date: 2015-01-14T14:40:14Z use tableIdentifier in metastore commit 111331c8988ba98368e6b227b3dc50f52136a0b4 Author: scwf Date: 2015-01-14T15:29:31Z using tableidentfier when create table in hive commit 481ba95e5384dcadc439f0612f54d41bd9cea24a Author: scwf Date: 2015-01-14T15:41:07Z using tableidentifier in ddl also commit 3c1fcbbcf637b26b284e63984c5b1717072a5839 Author: scwf Date: 2015-01-14T15:56:48Z Merge branch 'tableIdentifier' into createtable commit e0d19600204c3a54ca9a6a959ccaaa1c0d7bcdca Author: scwf Date: 2015-01-15T14:14:53Z fix conflicts --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4943][SQL] Allow table name having dot ...
Github user alexliu68 commented on the pull request: https://github.com/apache/spark/pull/3941#issuecomment-69823910 I only change selection to use tableIdentifier for joining tables. I keep createTable no change to minimize API changes. createTable could also use tableIdentifier to create catalog/cluster/database level table, but I leave it for future development if we need it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4943][SQL] Allow table name having dot ...
Github user yhuai commented on the pull request: https://github.com/apache/spark/pull/3941#issuecomment-69818976 @alexliu68 Is there any particular reason that `org.apache.spark.sql.hive.HiveMetastoreCatalog#createTable` does not take a `tableIdentifier: Seq[String]`? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4943][SQL] Allow table name having dot ...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/3941 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4943][SQL] Allow table name having dot ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3941#issuecomment-69439432 [Test build #25356 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25356/consoleFull) for PR 3941 at commit [`343ae27`](https://github.com/apache/spark/commit/343ae27959bcccd20b7360c9a050eb297a181e14). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4943][SQL] Allow table name having dot ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3941#issuecomment-69439434 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25356/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4943][SQL] Allow table name having dot ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3941#issuecomment-69435983 [Test build #25356 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25356/consoleFull) for PR 3941 at commit [`343ae27`](https://github.com/apache/spark/commit/343ae27959bcccd20b7360c9a050eb297a181e14). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4943][SQL] Allow table name having dot ...
Github user alexliu68 commented on a diff in the pull request:
https://github.com/apache/spark/pull/3941#discussion_r22755736
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Catalog.scala
---
@@ -115,43 +101,41 @@ class SimpleCatalog(val caseSensitive: Boolean)
extends Catalog {
trait OverrideCatalog extends Catalog {
// TODO: This doesn't work when the database changes...
- val overrides = new mutable.HashMap[(Option[String],String),
LogicalPlan]()
+ val overrides = new mutable.HashMap[String, LogicalPlan]()
--- End diff --
restore it to (Option[String],String)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4943][SQL] Allow table name having dot ...
Github user alexliu68 commented on a diff in the pull request:
https://github.com/apache/spark/pull/3941#discussion_r22755728
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/SqlParser.scala ---
@@ -178,10 +178,23 @@ class SqlParser extends AbstractSparkSQLParser {
joinedRelation | relationFactor
protected lazy val relationFactor: Parser[LogicalPlan] =
-( ident ~ (opt(AS) ~> opt(ident)) ^^ {
-case tableName ~ alias => UnresolvedRelation(None, tableName,
alias)
+(
+ ident ~ ("." ~> ident) ~ ("." ~> ident) ~ ("." ~> ident) ~ (opt(AS)
~> opt(ident)) ^^ {
+case reserveName1 ~ reserveName2 ~ dbName ~ tableName ~ alias =>
+ UnresolvedRelation(IndexedSeq(tableName, dbName, reserveName2,
reserveName1), alias)
}
-| ("(" ~> start <~ ")") ~ (AS.? ~> ident) ^^ { case s ~ a =>
Subquery(a, s) }
+ | ident ~ ("." ~> ident) ~ ("." ~> ident) ~ (opt(AS) ~> opt(ident))
^^ {
+case reserveName1 ~ dbName ~ tableName ~ alias =>
+ UnresolvedRelation(IndexedSeq(tableName, dbName, reserveName1),
alias)
+ }
+ | ident ~ ("." ~> ident) ~ (opt(AS) ~> opt(ident)) ^^ {
+ case dbName ~ tableName ~ alias =>
+UnresolvedRelation(IndexedSeq(tableName, dbName), alias)
+}
+ | ident ~ (opt(AS) ~> opt(ident)) ^^ {
+ case tableName ~ alias =>
UnresolvedRelation(IndexedSeq(tableName), alias)
--- End diff --
I change it to rep1sep(ident, ".")
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4943][SQL] Allow table name having dot ...
Github user alexliu68 commented on a diff in the pull request:
https://github.com/apache/spark/pull/3941#discussion_r22746062
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Catalog.scala
---
@@ -28,77 +28,63 @@ trait Catalog {
def caseSensitive: Boolean
- def tableExists(db: Option[String], tableName: String): Boolean
+ def tableExists(tableIdentifier: Seq[String]): Boolean
def lookupRelation(
-databaseName: Option[String],
-tableName: String,
-alias: Option[String] = None): LogicalPlan
+ tableIdentifier: Seq[String],
+ alias: Option[String] = None): LogicalPlan
- def registerTable(databaseName: Option[String], tableName: String, plan:
LogicalPlan): Unit
+ def registerTable(tableIdentifier: Seq[String], plan: LogicalPlan): Unit
- def unregisterTable(databaseName: Option[String], tableName: String):
Unit
+ def unregisterTable(tableIdentifier: Seq[String]): Unit
def unregisterAllTables(): Unit
- protected def processDatabaseAndTableName(
- databaseName: Option[String],
- tableName: String): (Option[String], String) = {
+ protected def processTableIdentifier(tableIdentifier: Seq[String]):
+ Seq[String] = {
if (!caseSensitive) {
- (databaseName.map(_.toLowerCase), tableName.toLowerCase)
+ tableIdentifier.map(_.toLowerCase)
} else {
- (databaseName, tableName)
+ tableIdentifier
}
}
- protected def processDatabaseAndTableName(
- databaseName: String,
- tableName: String): (String, String) = {
-if (!caseSensitive) {
- (databaseName.toLowerCase, tableName.toLowerCase)
-} else {
- (databaseName, tableName)
-}
- }
}
class SimpleCatalog(val caseSensitive: Boolean) extends Catalog {
val tables = new mutable.HashMap[String, LogicalPlan]()
override def registerTable(
- databaseName: Option[String],
- tableName: String,
+ tableIdentifier: Seq[String],
plan: LogicalPlan): Unit = {
-val (dbName, tblName) = processDatabaseAndTableName(databaseName,
tableName)
-tables += ((tblName, plan))
+val tableIdent = processTableIdentifier(tableIdentifier)
+tables += ((tableIdent.mkString("."), plan))
}
- override def unregisterTable(
- databaseName: Option[String],
- tableName: String) = {
-val (dbName, tblName) = processDatabaseAndTableName(databaseName,
tableName)
-tables -= tblName
+ override def unregisterTable(tableIdentifier: Seq[String]) = {
+val tableIdent = processTableIdentifier(tableIdentifier)
+tables -= tableIdent.mkString(".")
}
override def unregisterAllTables() = {
tables.clear()
}
- override def tableExists(db: Option[String], tableName: String): Boolean
= {
-val (dbName, tblName) = processDatabaseAndTableName(db, tableName)
-tables.get(tblName) match {
+ override def tableExists(tableIdentifier: Seq[String]): Boolean = {
+val tableIdent = processTableIdentifier(tableIdentifier)
+tables.get(tableIdent.mkString(".")) match {
case Some(_) => true
case None => false
}
}
override def lookupRelation(
- databaseName: Option[String],
- tableName: String,
+ tableIdentifier: Seq[String],
alias: Option[String] = None): LogicalPlan = {
-val (dbName, tblName) = processDatabaseAndTableName(databaseName,
tableName)
-val table = tables.getOrElse(tblName, sys.error(s"Table Not Found:
$tableName"))
-val tableWithQualifiers = Subquery(tblName, table)
+val tableIdent = processTableIdentifier(tableIdentifier)
+val tableFullName = tableIdent.mkString(".")
+val table = tables.getOrElse(tableFullName, sys.error(s"Table Not
Found: $tableFullName"))
+val tableWithQualifiers = Subquery(tableIdent.head, table)
--- End diff --
agreed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4943][SQL] Allow table name having dot ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3941#issuecomment-69404851 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25340/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4943][SQL] Allow table name having dot ...
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/3941#discussion_r22745717
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Catalog.scala
---
@@ -28,77 +28,63 @@ trait Catalog {
def caseSensitive: Boolean
- def tableExists(db: Option[String], tableName: String): Boolean
+ def tableExists(tableIdentifier: Seq[String]): Boolean
def lookupRelation(
-databaseName: Option[String],
-tableName: String,
-alias: Option[String] = None): LogicalPlan
+ tableIdentifier: Seq[String],
+ alias: Option[String] = None): LogicalPlan
- def registerTable(databaseName: Option[String], tableName: String, plan:
LogicalPlan): Unit
+ def registerTable(tableIdentifier: Seq[String], plan: LogicalPlan): Unit
- def unregisterTable(databaseName: Option[String], tableName: String):
Unit
+ def unregisterTable(tableIdentifier: Seq[String]): Unit
def unregisterAllTables(): Unit
- protected def processDatabaseAndTableName(
- databaseName: Option[String],
- tableName: String): (Option[String], String) = {
+ protected def processTableIdentifier(tableIdentifier: Seq[String]):
+ Seq[String] = {
if (!caseSensitive) {
- (databaseName.map(_.toLowerCase), tableName.toLowerCase)
+ tableIdentifier.map(_.toLowerCase)
} else {
- (databaseName, tableName)
+ tableIdentifier
}
}
- protected def processDatabaseAndTableName(
- databaseName: String,
- tableName: String): (String, String) = {
-if (!caseSensitive) {
- (databaseName.toLowerCase, tableName.toLowerCase)
-} else {
- (databaseName, tableName)
-}
- }
}
class SimpleCatalog(val caseSensitive: Boolean) extends Catalog {
val tables = new mutable.HashMap[String, LogicalPlan]()
override def registerTable(
- databaseName: Option[String],
- tableName: String,
+ tableIdentifier: Seq[String],
plan: LogicalPlan): Unit = {
-val (dbName, tblName) = processDatabaseAndTableName(databaseName,
tableName)
-tables += ((tblName, plan))
+val tableIdent = processTableIdentifier(tableIdentifier)
+tables += ((tableIdent.mkString("."), plan))
}
- override def unregisterTable(
- databaseName: Option[String],
- tableName: String) = {
-val (dbName, tblName) = processDatabaseAndTableName(databaseName,
tableName)
-tables -= tblName
+ override def unregisterTable(tableIdentifier: Seq[String]) = {
+val tableIdent = processTableIdentifier(tableIdentifier)
+tables -= tableIdent.mkString(".")
}
override def unregisterAllTables() = {
tables.clear()
}
- override def tableExists(db: Option[String], tableName: String): Boolean
= {
-val (dbName, tblName) = processDatabaseAndTableName(db, tableName)
-tables.get(tblName) match {
+ override def tableExists(tableIdentifier: Seq[String]): Boolean = {
+val tableIdent = processTableIdentifier(tableIdentifier)
+tables.get(tableIdent.mkString(".")) match {
case Some(_) => true
case None => false
}
}
override def lookupRelation(
- databaseName: Option[String],
- tableName: String,
+ tableIdentifier: Seq[String],
alias: Option[String] = None): LogicalPlan = {
-val (dbName, tblName) = processDatabaseAndTableName(databaseName,
tableName)
-val table = tables.getOrElse(tblName, sys.error(s"Table Not Found:
$tableName"))
-val tableWithQualifiers = Subquery(tblName, table)
+val tableIdent = processTableIdentifier(tableIdentifier)
+val tableFullName = tableIdent.mkString(".")
+val table = tables.getOrElse(tableFullName, sys.error(s"Table Not
Found: $tableFullName"))
+val tableWithQualifiers = Subquery(tableIdent.head, table)
--- End diff --
That does not seem worth the confusion of having it in reverse order. Also
looking at that code more closely, we would silently discard extra part of the
table identifier right? Seems like we should explicitly handle the cases where
there are 1, 2, and more elements of the table identifier.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-
[GitHub] spark pull request: [SPARK-4943][SQL] Allow table name having dot ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3941#issuecomment-69404842 [Test build #25340 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25340/consoleFull) for PR 3941 at commit [`29e5e55`](https://github.com/apache/spark/commit/29e5e55d39af6abe49eea8ef97b0d394caa8f5ab). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4943][SQL] Allow table name having dot ...
Github user alexliu68 commented on a diff in the pull request:
https://github.com/apache/spark/pull/3941#discussion_r22745569
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Catalog.scala
---
@@ -28,77 +28,63 @@ trait Catalog {
def caseSensitive: Boolean
- def tableExists(db: Option[String], tableName: String): Boolean
+ def tableExists(tableIdentifier: Seq[String]): Boolean
def lookupRelation(
-databaseName: Option[String],
-tableName: String,
-alias: Option[String] = None): LogicalPlan
+ tableIdentifier: Seq[String],
+ alias: Option[String] = None): LogicalPlan
- def registerTable(databaseName: Option[String], tableName: String, plan:
LogicalPlan): Unit
+ def registerTable(tableIdentifier: Seq[String], plan: LogicalPlan): Unit
- def unregisterTable(databaseName: Option[String], tableName: String):
Unit
+ def unregisterTable(tableIdentifier: Seq[String]): Unit
def unregisterAllTables(): Unit
- protected def processDatabaseAndTableName(
- databaseName: Option[String],
- tableName: String): (Option[String], String) = {
+ protected def processTableIdentifier(tableIdentifier: Seq[String]):
+ Seq[String] = {
if (!caseSensitive) {
- (databaseName.map(_.toLowerCase), tableName.toLowerCase)
+ tableIdentifier.map(_.toLowerCase)
} else {
- (databaseName, tableName)
+ tableIdentifier
}
}
- protected def processDatabaseAndTableName(
- databaseName: String,
- tableName: String): (String, String) = {
-if (!caseSensitive) {
- (databaseName.toLowerCase, tableName.toLowerCase)
-} else {
- (databaseName, tableName)
-}
- }
}
class SimpleCatalog(val caseSensitive: Boolean) extends Catalog {
val tables = new mutable.HashMap[String, LogicalPlan]()
override def registerTable(
- databaseName: Option[String],
- tableName: String,
+ tableIdentifier: Seq[String],
plan: LogicalPlan): Unit = {
-val (dbName, tblName) = processDatabaseAndTableName(databaseName,
tableName)
-tables += ((tblName, plan))
+val tableIdent = processTableIdentifier(tableIdentifier)
+tables += ((tableIdent.mkString("."), plan))
}
- override def unregisterTable(
- databaseName: Option[String],
- tableName: String) = {
-val (dbName, tblName) = processDatabaseAndTableName(databaseName,
tableName)
-tables -= tblName
+ override def unregisterTable(tableIdentifier: Seq[String]) = {
+val tableIdent = processTableIdentifier(tableIdentifier)
+tables -= tableIdent.mkString(".")
}
override def unregisterAllTables() = {
tables.clear()
}
- override def tableExists(db: Option[String], tableName: String): Boolean
= {
-val (dbName, tblName) = processDatabaseAndTableName(db, tableName)
-tables.get(tblName) match {
+ override def tableExists(tableIdentifier: Seq[String]): Boolean = {
+val tableIdent = processTableIdentifier(tableIdentifier)
+tables.get(tableIdent.mkString(".")) match {
case Some(_) => true
case None => false
}
}
override def lookupRelation(
- databaseName: Option[String],
- tableName: String,
+ tableIdentifier: Seq[String],
alias: Option[String] = None): LogicalPlan = {
-val (dbName, tblName) = processDatabaseAndTableName(databaseName,
tableName)
-val table = tables.getOrElse(tblName, sys.error(s"Table Not Found:
$tableName"))
-val tableWithQualifiers = Subquery(tblName, table)
+val tableIdent = processTableIdentifier(tableIdentifier)
+val tableFullName = tableIdent.mkString(".")
+val table = tables.getOrElse(tableFullName, sys.error(s"Table Not
Found: $tableFullName"))
+val tableWithQualifiers = Subquery(tableIdent.head, table)
--- End diff --
By reversed order, we know database name will be at 2nd, so it's easy to
access it by using lift(1)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4943][SQL] Allow table name having dot ...
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/3941#issuecomment-69403750 Its minor, but it seems unlikely that it it noticeably faster in this instance. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4943][SQL] Allow table name having dot ...
Github user alexliu68 commented on the pull request: https://github.com/apache/spark/pull/3941#issuecomment-69403586 I use IndexedSeq instead Seq for IndexedSeq is faster to access element by lift. I can revert them back to Seq. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4943][SQL] Allow table name having dot ...
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/3941#discussion_r22744822
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Catalog.scala
---
@@ -28,77 +28,63 @@ trait Catalog {
def caseSensitive: Boolean
- def tableExists(db: Option[String], tableName: String): Boolean
+ def tableExists(tableIdentifier: Seq[String]): Boolean
def lookupRelation(
-databaseName: Option[String],
-tableName: String,
-alias: Option[String] = None): LogicalPlan
+ tableIdentifier: Seq[String],
+ alias: Option[String] = None): LogicalPlan
- def registerTable(databaseName: Option[String], tableName: String, plan:
LogicalPlan): Unit
+ def registerTable(tableIdentifier: Seq[String], plan: LogicalPlan): Unit
- def unregisterTable(databaseName: Option[String], tableName: String):
Unit
+ def unregisterTable(tableIdentifier: Seq[String]): Unit
def unregisterAllTables(): Unit
- protected def processDatabaseAndTableName(
- databaseName: Option[String],
- tableName: String): (Option[String], String) = {
+ protected def processTableIdentifier(tableIdentifier: Seq[String]):
+ Seq[String] = {
if (!caseSensitive) {
- (databaseName.map(_.toLowerCase), tableName.toLowerCase)
+ tableIdentifier.map(_.toLowerCase)
} else {
- (databaseName, tableName)
+ tableIdentifier
}
}
- protected def processDatabaseAndTableName(
- databaseName: String,
- tableName: String): (String, String) = {
-if (!caseSensitive) {
- (databaseName.toLowerCase, tableName.toLowerCase)
-} else {
- (databaseName, tableName)
-}
- }
}
class SimpleCatalog(val caseSensitive: Boolean) extends Catalog {
val tables = new mutable.HashMap[String, LogicalPlan]()
override def registerTable(
- databaseName: Option[String],
- tableName: String,
+ tableIdentifier: Seq[String],
plan: LogicalPlan): Unit = {
-val (dbName, tblName) = processDatabaseAndTableName(databaseName,
tableName)
-tables += ((tblName, plan))
+val tableIdent = processTableIdentifier(tableIdentifier)
+tables += ((tableIdent.mkString("."), plan))
}
- override def unregisterTable(
- databaseName: Option[String],
- tableName: String) = {
-val (dbName, tblName) = processDatabaseAndTableName(databaseName,
tableName)
-tables -= tblName
+ override def unregisterTable(tableIdentifier: Seq[String]) = {
+val tableIdent = processTableIdentifier(tableIdentifier)
+tables -= tableIdent.mkString(".")
}
override def unregisterAllTables() = {
tables.clear()
}
- override def tableExists(db: Option[String], tableName: String): Boolean
= {
-val (dbName, tblName) = processDatabaseAndTableName(db, tableName)
-tables.get(tblName) match {
+ override def tableExists(tableIdentifier: Seq[String]): Boolean = {
+val tableIdent = processTableIdentifier(tableIdentifier)
+tables.get(tableIdent.mkString(".")) match {
case Some(_) => true
case None => false
}
}
override def lookupRelation(
- databaseName: Option[String],
- tableName: String,
+ tableIdentifier: Seq[String],
alias: Option[String] = None): LogicalPlan = {
-val (dbName, tblName) = processDatabaseAndTableName(databaseName,
tableName)
-val table = tables.getOrElse(tblName, sys.error(s"Table Not Found:
$tableName"))
-val tableWithQualifiers = Subquery(tblName, table)
+val tableIdent = processTableIdentifier(tableIdentifier)
+val tableFullName = tableIdent.mkString(".")
+val table = tables.getOrElse(tableFullName, sys.error(s"Table Not
Found: $tableFullName"))
+val tableWithQualifiers = Subquery(tableIdent.head, table)
--- End diff --
Oh, sorry that was not obvious to me, and seems pretty confusing. Why not
store it in the order the user gave it to you?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4943][SQL] Allow table name having dot ...
Github user alexliu68 commented on a diff in the pull request:
https://github.com/apache/spark/pull/3941#discussion_r22744829
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala ---
@@ -251,6 +257,26 @@ private[hive] class HiveMetastoreCatalog(hive:
HiveContext) extends Catalog with
}
}
+ protected def processDatabaseAndTableName(
+ databaseName: Option[String],
+ tableName: String): (Option[String], String) = {
+if (!caseSensitive) {
+ (databaseName.map(_.toLowerCase), tableName.toLowerCase)
+} else {
+ (databaseName, tableName)
+}
+ }
+
+ protected def processDatabaseAndTableName(
--- End diff --
It's only used by hive table creation
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4943][SQL] Allow table name having dot ...
Github user alexliu68 commented on a diff in the pull request:
https://github.com/apache/spark/pull/3941#discussion_r22744754
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveQl.scala ---
@@ -386,6 +386,13 @@ private[hive] object HiveQl {
(db, tableName)
}
+ protected def extractTableIdent(tableNameParts: Node):
IndexedSeq[String] = {
+tableNameParts.getChildren.map { case Token(part, Nil) =>
cleanIdentifier(part) } match {
+ case Seq(tableOnly) => IndexedSeq(tableOnly)
+ case Seq(databaseName, table) => IndexedSeq(table, databaseName)
--- End diff --
agreed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4943][SQL] Allow table name having dot ...
Github user alexliu68 commented on a diff in the pull request:
https://github.com/apache/spark/pull/3941#discussion_r22744744
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Catalog.scala
---
@@ -28,77 +28,63 @@ trait Catalog {
def caseSensitive: Boolean
- def tableExists(db: Option[String], tableName: String): Boolean
+ def tableExists(tableIdentifier: Seq[String]): Boolean
def lookupRelation(
-databaseName: Option[String],
-tableName: String,
-alias: Option[String] = None): LogicalPlan
+ tableIdentifier: Seq[String],
+ alias: Option[String] = None): LogicalPlan
- def registerTable(databaseName: Option[String], tableName: String, plan:
LogicalPlan): Unit
+ def registerTable(tableIdentifier: Seq[String], plan: LogicalPlan): Unit
- def unregisterTable(databaseName: Option[String], tableName: String):
Unit
+ def unregisterTable(tableIdentifier: Seq[String]): Unit
def unregisterAllTables(): Unit
- protected def processDatabaseAndTableName(
- databaseName: Option[String],
- tableName: String): (Option[String], String) = {
+ protected def processTableIdentifier(tableIdentifier: Seq[String]):
+ Seq[String] = {
if (!caseSensitive) {
- (databaseName.map(_.toLowerCase), tableName.toLowerCase)
+ tableIdentifier.map(_.toLowerCase)
} else {
- (databaseName, tableName)
+ tableIdentifier
}
}
- protected def processDatabaseAndTableName(
- databaseName: String,
- tableName: String): (String, String) = {
-if (!caseSensitive) {
- (databaseName.toLowerCase, tableName.toLowerCase)
-} else {
- (databaseName, tableName)
-}
- }
}
class SimpleCatalog(val caseSensitive: Boolean) extends Catalog {
val tables = new mutable.HashMap[String, LogicalPlan]()
override def registerTable(
- databaseName: Option[String],
- tableName: String,
+ tableIdentifier: Seq[String],
plan: LogicalPlan): Unit = {
-val (dbName, tblName) = processDatabaseAndTableName(databaseName,
tableName)
-tables += ((tblName, plan))
+val tableIdent = processTableIdentifier(tableIdentifier)
+tables += ((tableIdent.mkString("."), plan))
}
- override def unregisterTable(
- databaseName: Option[String],
- tableName: String) = {
-val (dbName, tblName) = processDatabaseAndTableName(databaseName,
tableName)
-tables -= tblName
+ override def unregisterTable(tableIdentifier: Seq[String]) = {
+val tableIdent = processTableIdentifier(tableIdentifier)
+tables -= tableIdent.mkString(".")
}
override def unregisterAllTables() = {
tables.clear()
}
- override def tableExists(db: Option[String], tableName: String): Boolean
= {
-val (dbName, tblName) = processDatabaseAndTableName(db, tableName)
-tables.get(tblName) match {
+ override def tableExists(tableIdentifier: Seq[String]): Boolean = {
+val tableIdent = processTableIdentifier(tableIdentifier)
+tables.get(tableIdent.mkString(".")) match {
case Some(_) => true
case None => false
}
}
override def lookupRelation(
- databaseName: Option[String],
- tableName: String,
+ tableIdentifier: Seq[String],
alias: Option[String] = None): LogicalPlan = {
-val (dbName, tblName) = processDatabaseAndTableName(databaseName,
tableName)
-val table = tables.getOrElse(tblName, sys.error(s"Table Not Found:
$tableName"))
-val tableWithQualifiers = Subquery(tblName, table)
+val tableIdent = processTableIdentifier(tableIdentifier)
+val tableFullName = tableIdent.mkString(".")
+val table = tables.getOrElse(tableFullName, sys.error(s"Table Not
Found: $tableFullName"))
+val tableWithQualifiers = Subquery(tableIdent.head, table)
--- End diff --
I store the seq in reversed order, so table name is at head.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4943][SQL] Allow table name having dot ...
Github user alexliu68 commented on a diff in the pull request:
https://github.com/apache/spark/pull/3941#discussion_r22744697
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Catalog.scala
---
@@ -28,77 +28,63 @@ trait Catalog {
def caseSensitive: Boolean
- def tableExists(db: Option[String], tableName: String): Boolean
+ def tableExists(tableIdentifier: Seq[String]): Boolean
def lookupRelation(
-databaseName: Option[String],
-tableName: String,
-alias: Option[String] = None): LogicalPlan
+ tableIdentifier: Seq[String],
+ alias: Option[String] = None): LogicalPlan
--- End diff --
agreed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4943][SQL] Allow table name having dot ...
Github user alexliu68 commented on a diff in the pull request:
https://github.com/apache/spark/pull/3941#discussion_r22744696
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Catalog.scala
---
@@ -28,77 +28,63 @@ trait Catalog {
def caseSensitive: Boolean
- def tableExists(db: Option[String], tableName: String): Boolean
+ def tableExists(tableIdentifier: Seq[String]): Boolean
def lookupRelation(
-databaseName: Option[String],
-tableName: String,
-alias: Option[String] = None): LogicalPlan
+ tableIdentifier: Seq[String],
+ alias: Option[String] = None): LogicalPlan
- def registerTable(databaseName: Option[String], tableName: String, plan:
LogicalPlan): Unit
+ def registerTable(tableIdentifier: Seq[String], plan: LogicalPlan): Unit
- def unregisterTable(databaseName: Option[String], tableName: String):
Unit
+ def unregisterTable(tableIdentifier: Seq[String]): Unit
def unregisterAllTables(): Unit
- protected def processDatabaseAndTableName(
- databaseName: Option[String],
- tableName: String): (Option[String], String) = {
+ protected def processTableIdentifier(tableIdentifier: Seq[String]):
+ Seq[String] = {
--- End diff --
agreed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4943][SQL] Allow table name having dot ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3941#issuecomment-69395747 [Test build #25340 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25340/consoleFull) for PR 3941 at commit [`29e5e55`](https://github.com/apache/spark/commit/29e5e55d39af6abe49eea8ef97b0d394caa8f5ab). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4943][SQL] Allow table name having dot ...
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/3941#issuecomment-69395646 ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4943][SQL] Allow table name having dot ...
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/3941#issuecomment-69394945 Also, can you fix the PR title to avoid truncating. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4943][SQL] Allow table name having dot ...
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/3941#issuecomment-69394919 Thanks for doing this cleanup! Overall I think this looks pretty good. I made a few small comments. One other very minor nit. It seems a little odd to me to be using `IndexedSeq` everywhere instead of just `Seq` or `:: Nil`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4943][SQL] Allow table name having dot ...
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/3941#discussion_r22741238
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala ---
@@ -57,18 +58,23 @@ private[hive] class HiveMetastoreCatalog(hive:
HiveContext) extends Catalog with
val caseSensitive: Boolean = false
- def tableExists(db: Option[String], tableName: String): Boolean = {
-val (databaseName, tblName) = processDatabaseAndTableName(
- db.getOrElse(hive.sessionState.getCurrentDatabase), tableName)
-client.getTable(databaseName, tblName, false) != null
+ def tableExists(tableIdentifier: Seq[String]): Boolean = {
+val tableIdent = processTableIdentifier(tableIdentifier)
+val (databaseName, tblName) =
+ (tableIdent.lift(1).getOrElse(hive.sessionState.getCurrentDatabase),
tableIdent.head)
+try {
+ client.getTable(databaseName, tblName) != null
+} catch {
+ case ie: InvalidTableException => false
+}
}
def lookupRelation(
- db: Option[String],
- tableName: String,
+ tableIdentifier: Seq[String],
alias: Option[String]): LogicalPlan = synchronized {
+val tableIdent = processTableIdentifier(tableIdentifier)
val (databaseName, tblName) =
-
processDatabaseAndTableName(db.getOrElse(hive.sessionState.getCurrentDatabase),
tableName)
+ (tableIdent.lift(1).getOrElse(hive.sessionState.getCurrentDatabase),
tableIdent.head)
--- End diff --
I did not know that you could `lift` a `Seq` :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4943][SQL] Allow table name having dot ...
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/3941#discussion_r22741182
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala ---
@@ -251,6 +257,26 @@ private[hive] class HiveMetastoreCatalog(hive:
HiveContext) extends Catalog with
}
}
+ protected def processDatabaseAndTableName(
+ databaseName: Option[String],
+ tableName: String): (Option[String], String) = {
+if (!caseSensitive) {
+ (databaseName.map(_.toLowerCase), tableName.toLowerCase)
+} else {
+ (databaseName, tableName)
+}
+ }
+
+ protected def processDatabaseAndTableName(
--- End diff --
Are either of these functions used anymore?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4943][SQL] Allow table name having dot ...
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/3941#discussion_r22741136
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveQl.scala ---
@@ -386,6 +386,13 @@ private[hive] object HiveQl {
(db, tableName)
}
+ protected def extractTableIdent(tableNameParts: Node):
IndexedSeq[String] = {
+tableNameParts.getChildren.map { case Token(part, Nil) =>
cleanIdentifier(part) } match {
+ case Seq(tableOnly) => IndexedSeq(tableOnly)
+ case Seq(databaseName, table) => IndexedSeq(table, databaseName)
--- End diff --
I'd add a default case here that does something like `case other =>
sys.error(s"Hive only supports tables names like 'tableName' or
'databaseName.tableName', found '$other'")`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4943][SQL] Allow table name having dot ...
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/3941#discussion_r22740772
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Catalog.scala
---
@@ -28,77 +28,63 @@ trait Catalog {
def caseSensitive: Boolean
- def tableExists(db: Option[String], tableName: String): Boolean
+ def tableExists(tableIdentifier: Seq[String]): Boolean
def lookupRelation(
-databaseName: Option[String],
-tableName: String,
-alias: Option[String] = None): LogicalPlan
+ tableIdentifier: Seq[String],
+ alias: Option[String] = None): LogicalPlan
- def registerTable(databaseName: Option[String], tableName: String, plan:
LogicalPlan): Unit
+ def registerTable(tableIdentifier: Seq[String], plan: LogicalPlan): Unit
- def unregisterTable(databaseName: Option[String], tableName: String):
Unit
+ def unregisterTable(tableIdentifier: Seq[String]): Unit
def unregisterAllTables(): Unit
- protected def processDatabaseAndTableName(
- databaseName: Option[String],
- tableName: String): (Option[String], String) = {
+ protected def processTableIdentifier(tableIdentifier: Seq[String]):
+ Seq[String] = {
if (!caseSensitive) {
- (databaseName.map(_.toLowerCase), tableName.toLowerCase)
+ tableIdentifier.map(_.toLowerCase)
} else {
- (databaseName, tableName)
+ tableIdentifier
}
}
- protected def processDatabaseAndTableName(
- databaseName: String,
- tableName: String): (String, String) = {
-if (!caseSensitive) {
- (databaseName.toLowerCase, tableName.toLowerCase)
-} else {
- (databaseName, tableName)
-}
- }
}
class SimpleCatalog(val caseSensitive: Boolean) extends Catalog {
val tables = new mutable.HashMap[String, LogicalPlan]()
override def registerTable(
- databaseName: Option[String],
- tableName: String,
+ tableIdentifier: Seq[String],
plan: LogicalPlan): Unit = {
-val (dbName, tblName) = processDatabaseAndTableName(databaseName,
tableName)
-tables += ((tblName, plan))
+val tableIdent = processTableIdentifier(tableIdentifier)
+tables += ((tableIdent.mkString("."), plan))
}
- override def unregisterTable(
- databaseName: Option[String],
- tableName: String) = {
-val (dbName, tblName) = processDatabaseAndTableName(databaseName,
tableName)
-tables -= tblName
+ override def unregisterTable(tableIdentifier: Seq[String]) = {
+val tableIdent = processTableIdentifier(tableIdentifier)
+tables -= tableIdent.mkString(".")
}
override def unregisterAllTables() = {
tables.clear()
}
- override def tableExists(db: Option[String], tableName: String): Boolean
= {
-val (dbName, tblName) = processDatabaseAndTableName(db, tableName)
-tables.get(tblName) match {
+ override def tableExists(tableIdentifier: Seq[String]): Boolean = {
+val tableIdent = processTableIdentifier(tableIdentifier)
+tables.get(tableIdent.mkString(".")) match {
case Some(_) => true
case None => false
}
}
override def lookupRelation(
- databaseName: Option[String],
- tableName: String,
+ tableIdentifier: Seq[String],
alias: Option[String] = None): LogicalPlan = {
-val (dbName, tblName) = processDatabaseAndTableName(databaseName,
tableName)
-val table = tables.getOrElse(tblName, sys.error(s"Table Not Found:
$tableName"))
-val tableWithQualifiers = Subquery(tblName, table)
+val tableIdent = processTableIdentifier(tableIdentifier)
+val tableFullName = tableIdent.mkString(".")
+val table = tables.getOrElse(tableFullName, sys.error(s"Table Not
Found: $tableFullName"))
+val tableWithQualifiers = Subquery(tableIdent.head, table)
--- End diff --
`tableIdent.head`? Should it be `last`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4943][SQL] Allow table name having dot ...
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/3941#discussion_r22740657
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Catalog.scala
---
@@ -28,77 +28,63 @@ trait Catalog {
def caseSensitive: Boolean
- def tableExists(db: Option[String], tableName: String): Boolean
+ def tableExists(tableIdentifier: Seq[String]): Boolean
def lookupRelation(
-databaseName: Option[String],
-tableName: String,
-alias: Option[String] = None): LogicalPlan
+ tableIdentifier: Seq[String],
+ alias: Option[String] = None): LogicalPlan
--- End diff --
Indent wrapped arguments 4 spaces.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4943][SQL] Allow table name having dot ...
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/3941#discussion_r22740618
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Catalog.scala
---
@@ -28,77 +28,63 @@ trait Catalog {
def caseSensitive: Boolean
- def tableExists(db: Option[String], tableName: String): Boolean
+ def tableExists(tableIdentifier: Seq[String]): Boolean
def lookupRelation(
-databaseName: Option[String],
-tableName: String,
-alias: Option[String] = None): LogicalPlan
+ tableIdentifier: Seq[String],
+ alias: Option[String] = None): LogicalPlan
- def registerTable(databaseName: Option[String], tableName: String, plan:
LogicalPlan): Unit
+ def registerTable(tableIdentifier: Seq[String], plan: LogicalPlan): Unit
- def unregisterTable(databaseName: Option[String], tableName: String):
Unit
+ def unregisterTable(tableIdentifier: Seq[String]): Unit
def unregisterAllTables(): Unit
- protected def processDatabaseAndTableName(
- databaseName: Option[String],
- tableName: String): (Option[String], String) = {
+ protected def processTableIdentifier(tableIdentifier: Seq[String]):
+ Seq[String] = {
--- End diff --
I don't think this needs to be wrapped, does it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4943][SQL] Allow table name having dot ...
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/3941#discussion_r22740414
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Catalog.scala
---
@@ -115,43 +101,41 @@ class SimpleCatalog(val caseSensitive: Boolean)
extends Catalog {
trait OverrideCatalog extends Catalog {
// TODO: This doesn't work when the database changes...
- val overrides = new mutable.HashMap[(Option[String],String),
LogicalPlan]()
+ val overrides = new mutable.HashMap[String, LogicalPlan]()
--- End diff --
Why collapse this to a `String`? What about a table with a name like the
following:
```
SELECT * FROM `database`.`table.name`
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4943][SQL] Allow table name having dot ...
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/3941#discussion_r22740262
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/SqlParser.scala ---
@@ -178,10 +178,23 @@ class SqlParser extends AbstractSparkSQLParser {
joinedRelation | relationFactor
protected lazy val relationFactor: Parser[LogicalPlan] =
-( ident ~ (opt(AS) ~> opt(ident)) ^^ {
-case tableName ~ alias => UnresolvedRelation(None, tableName,
alias)
+(
+ ident ~ ("." ~> ident) ~ ("." ~> ident) ~ ("." ~> ident) ~ (opt(AS)
~> opt(ident)) ^^ {
+case reserveName1 ~ reserveName2 ~ dbName ~ tableName ~ alias =>
+ UnresolvedRelation(IndexedSeq(tableName, dbName, reserveName2,
reserveName1), alias)
}
-| ("(" ~> start <~ ")") ~ (AS.? ~> ident) ^^ { case s ~ a =>
Subquery(a, s) }
+ | ident ~ ("." ~> ident) ~ ("." ~> ident) ~ (opt(AS) ~> opt(ident))
^^ {
+case reserveName1 ~ dbName ~ tableName ~ alias =>
+ UnresolvedRelation(IndexedSeq(tableName, dbName, reserveName1),
alias)
+ }
+ | ident ~ ("." ~> ident) ~ (opt(AS) ~> opt(ident)) ^^ {
+ case dbName ~ tableName ~ alias =>
+UnresolvedRelation(IndexedSeq(tableName, dbName), alias)
+}
+ | ident ~ (opt(AS) ~> opt(ident)) ^^ {
+ case tableName ~ alias =>
UnresolvedRelation(IndexedSeq(tableName), alias)
--- End diff --
I think this should probably just be a single rule. With something like:
`repsep(ident, ",")` for the table identifier.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4943][SQL] Allow table name having dot ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3941#issuecomment-69137134 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4943][SQL] Allow table name having dot ...
GitHub user alexliu68 opened a pull request: https://github.com/apache/spark/pull/3941 [SPARK-4943][SQL] Allow table name having dot to support db/catalog ... ...r The pull only fixes the parsing error and changes API to use tableIdentifier. Joining different catalog datasource related change is not done in this pull. You can merge this pull request into a Git repository by running: $ git pull https://github.com/alexliu68/spark SPARK-SQL-4943-3 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/3941.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3941 commit 365299739c92132cd34110a00e860e218e7ca9c6 Author: Alex Liu Date: 2015-01-08T05:09:17Z [SPARK-4943][SQL] Allow table name having dot to support db/catalog ... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4943][SQL] Allow table name having dot ...
Github user alexliu68 commented on the pull request: https://github.com/apache/spark/pull/3848#issuecomment-68470193 close the pull for further discussion in Jira ticket. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4943][SQL] Allow table name having dot ...
Github user alexliu68 closed the pull request at: https://github.com/apache/spark/pull/3848 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4943][SQL] Allow table name having dot ...
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/3848#discussion_r22393319
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/SqlParser.scala ---
@@ -178,7 +178,15 @@ class SqlParser extends AbstractSparkSQLParser {
joinedRelation | relationFactor
protected lazy val relationFactor: Parser[LogicalPlan] =
-( ident ~ (opt(AS) ~> opt(ident)) ^^ {
+( ident ~ ("." ~> ident) ~ ("." ~> ident) ~ (opt(AS) ~> opt(ident)) ^^
{
+case clusterName ~ dbName ~ tableName ~ alias =>
+ UnresolvedRelation(None, clusterName + "." + dbName + "." +
tableName, alias)
--- End diff --
Okay, but there is a field called `databaseName` and you aren't using it,
and so this PR is putting the SQLParser and the HiveQL parser out of sync. I
think we should close this issue for now and discuss the requirements and
design on the JIRA.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4943][SQL] Allow table name having dot ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3848#issuecomment-68415108 [Test build #24918 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24918/consoleFull) for PR 3848 at commit [`00adf40`](https://github.com/apache/spark/commit/00adf40eb0c94fab8a24861dc1e79b17eda7612a). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4943][SQL] Allow table name having dot ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3848#issuecomment-68415109 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24918/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4943][SQL] Allow table name having dot ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3848#issuecomment-68411286 [Test build #24918 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24918/consoleFull) for PR 3848 at commit [`00adf40`](https://github.com/apache/spark/commit/00adf40eb0c94fab8a24861dc1e79b17eda7612a). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4943][SQL] Allow table name having dot ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3848#issuecomment-68410695 [Test build #24916 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24916/consoleFull) for PR 3848 at commit [`623cf91`](https://github.com/apache/spark/commit/623cf91d6fac52a2f6cdca74589145e502225066). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4943][SQL] Allow table name having dot ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3848#issuecomment-68410696 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24916/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4943][SQL] Allow table name having dot ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3848#issuecomment-68410615 [Test build #24916 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24916/consoleFull) for PR 3848 at commit [`623cf91`](https://github.com/apache/spark/commit/623cf91d6fac52a2f6cdca74589145e502225066). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4943][SQL] Allow table name having dot ...
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/3848#issuecomment-68410581 ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4943][SQL] Allow table name having dot ...
Github user alexliu68 commented on the pull request: https://github.com/apache/spark/pull/3848#issuecomment-68409413 I rebase the commit to simply fixing the parsing issue and concatenate cluster name, database name and table name into a single string with dot in between. Now it can parse [clusterName].[databaseName].[tableName] type full table name and pass it as tableName to catalog. The following example query works for Cassandra integration e.g. Select table.column from cluster.database.table I leave the refactoring to better support full table name to future work. Ideally Spark SQL should be able to join data across catalog, cluster, database and table. There are four levels of data joining, catalog level, cluster level, database level, and table level. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4943][SQL] Allow table name having dot ...
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/3848#issuecomment-68392502 I mean instead it would look like: ```scala case class UnresolvedRelation(tableName: Seq[String], alias: Option[String]) ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4943][SQL] Allow table name having dot ...
Github user alexliu68 commented on the pull request: https://github.com/apache/spark/pull/3848#issuecomment-68392112 Strings separated by space? It would work, let me test it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4943][SQL] Allow table name having dot ...
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/3848#issuecomment-68391743 What about a sequence of strings where each entry is some part of the table identifier? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4943][SQL] Allow table name having dot ...
Github user alexliu68 commented on the pull request: https://github.com/apache/spark/pull/3848#issuecomment-68390959 It can apply to any type of datasources. e.g. HBase, Oracle, MongoDB. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4943][SQL] Allow table name having dot ...
Github user alexliu68 commented on the pull request: https://github.com/apache/spark/pull/3848#issuecomment-68390702 It's not for a single system. It's more a general approach. e.g. you have many mySql clusters and each cluster has databases and each database has tables. Right now it can't parse table name having Dot. Let each system parse table name into cluster/database/table is a little risk for parsing error may occur if the Spark parser changes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4943][SQL] Allow table name having dot ...
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/3848#issuecomment-68390025 Our handling of table names definitely needs some work, but I don't think the right thing to do is to add yet another special case here that only applies to a single system. Perhaps the right thing to do is to keep the whole table identifier together as a single string and let various catalogs parse it as they want. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4943][SQL] Allow table name having dot ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3848#issuecomment-68389804 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4943][SQL] Allow table name having dot ...
Github user alexliu68 commented on the pull request: https://github.com/apache/spark/pull/3848#issuecomment-68389753 Support table full name in the format of .. so that we can join data from different tables and tables can be from different databases and clusters. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
