[GitHub] spark pull request: [SPARK-12573][SPARK-12574][SQL] Move SQL Parse...
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/10583#discussion_r48940817
--- Diff:
sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/parser/ParseUtils.java
---
@@ -0,0 +1,163 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.parser;
+
+import java.io.UnsupportedEncodingException;
+
+/**
+ * A couple of utility methods that help with parsing ASTs.
+ *
+ * Both methods in this class were take from the SemanticAnalyzer in Hive:
+ * ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
+ */
+public final class ParseUtils {
+ private ParseUtils() {
+super();
+ }
+
+ public static String charSetString(String charSetName, String
charSetString)
--- End diff --
I just checked. There aren't any? Bit strange...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12573][SPARK-12574][SQL] Move SQL Parse...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10583#issuecomment-169305716 **[Test build #48853 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48853/consoleFull)** for PR 10583 at commit [`3680d4c`](https://github.com/apache/spark/commit/3680d4c4179d63584a42f6d33bbe3c718fa3075d). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12573][SPARK-12574][SQL] Move SQL Parse...
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/10583#discussion_r48940364
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/CatalystQl.scala ---
@@ -0,0 +1,969 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst
+
+import java.sql.Date
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.analysis._
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.aggregate.Count
+import org.apache.spark.sql.catalyst.plans._
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.trees.CurrentOrigin
+import org.apache.spark.sql.catalyst.parser._
+import org.apache.spark.sql.types._
+import org.apache.spark.unsafe.types.CalendarInterval
+import org.apache.spark.util.random.RandomSampler
+
+/**
+ * This class translates a HQL String to a Catalyst [[LogicalPlan]] or
[[Expression]].
+ */
+private[sql] class CatalystQl(val conf: ParserConf = SimpleParserConf()) {
+ object Token {
+def unapply(node: ASTNode): Some[(String, List[ASTNode])] = {
+ CurrentOrigin.setPosition(node.line, node.positionInLine)
+ node.pattern
+}
+ }
+
+ // TODO improve the parse error - so we don't need this anymore.
+ val errorRegEx = "line (\\d+):(\\d+) (.*)".r
+
+ /**
+ * Returns the AST for the given SQL string.
+ */
+ protected def getAst(sql: String): ASTNode = ParseDriver.parse(sql, conf)
+
+ /** Creates LogicalPlan for a given HiveQL string. */
+ def createPlan(sql: String): LogicalPlan = {
+try {
+ createPlan(sql, ParseDriver.parse(sql, conf))
+} catch {
+ case pe: ParseException =>
+pe.getMessage match {
+ case errorRegEx(line, start, message) =>
+throw new AnalysisException(message, Some(line.toInt),
Some(start.toInt))
+ case otherMessage =>
+throw new AnalysisException(otherMessage)
+}
+ case e: MatchError => throw e
+ case e: Exception =>
+throw new AnalysisException(e.getMessage)
+ case e: NotImplementedError =>
+throw new AnalysisException(
+ s"""
+ |Unsupported language features in query: $sql
+ |${getAst(sql).treeString}
+ |$e
+ |${e.getStackTrace.head}
+ """.stripMargin)
+}
+ }
+
+ protected def createPlan(sql: String, tree: ASTNode): LogicalPlan =
nodeToPlan(tree)
+
+ def parseDdl(ddl: String): Seq[Attribute] = {
+val tree =
+ try {
+getAst(ddl)
+ } catch {
+case pe: ParseException =>
+ throw new RuntimeException(s"Failed to parse ddl: '$ddl'", pe)
+ }
+assert(tree.text == "TOK_CREATETABLE", "Only CREATE TABLE supported.")
+val tableOps = tree.children
+val colList = tableOps
+ .find(_.text == "TOK_TABCOLLIST")
+ .getOrElse(sys.error("No columnList!"))
+
+colList.children.map(nodeToAttribute)
+ }
+
+ protected def getClauses(
+ clauseNames: Seq[String],
+ nodeList: Seq[ASTNode]): Seq[Option[ASTNode]] = {
+var remainingNodes = nodeList
+val clauses = clauseNames.map { clauseName =>
+ val (matches, nonMatches) =
remainingNodes.partition(_.text.toUpperCase == clauseName)
+ remainingNodes = nonMatches ++ (if (matches.nonEmpty) matches.tail
else Nil)
+ matches.headOption
+}
+
+if (remainingNodes.nonEmpty) {
+ sys.error(
+s"""Unhandled clauses:
${remainingNodes.map(_.treeString).mkString("\n")}.
+|You are likely trying to use an unsupported Hive
feature..stripMargin)
+}
+clauses
+ }
+
+ protected def getClause(clauseName: String,
[GitHub] spark pull request: [SPARK-12573][SPARK-12574][SQL] Move SQL Parse...
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/10583#discussion_r48951342
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLConf.scala ---
@@ -451,6 +452,19 @@ private[spark] object SQLConf {
doc = "When true, we could use `datasource`.`path` as table in SQL
query"
)
+ val PARSER_SUPPORT_QUOTEDID =
booleanConf("spark.sql.parser.supportQuotedIdentifiers",
+defaultValue = Some(true),
+isPublic = false,
+doc = "Whether to use quoted identifier.\n false: default(past)
behavior. Implies only" +
+ "alphaNumeric and underscore are valid characters in identifiers.\n"
+
+ " true: implies column names can contain any character.")
+
+ val PARSER_SUPPORT_SQL11_RESERVED_KEYWORDS = booleanConf(
+"spark.sql.parser.supportSQL11ReservedKeywords",
+defaultValue = Some(false),
--- End diff --
Some of our out-dated Hive Compatibility Tests are using SQL11 reserved
keywords as identifiers. We should first fix those before we can set this flag
to true.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12573][SPARK-12574][SQL] Move SQL Parse...
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/10583#discussion_r48951594 --- Diff: dev/deps/spark-deps-hadoop-2.2 --- @@ -5,8 +5,7 @@ activation-1.1.jar akka-actor_2.10-2.3.11.jar akka-remote_2.10-2.3.11.jar akka-slf4j_2.10-2.3.11.jar -antlr-2.7.7.jar -antlr-runtime-3.4.jar +antlr-runtime-3.5.2.jar --- End diff -- I think we need to update LICENSE and NOTICE because of this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12573][SPARK-12574][SQL] Move SQL Parse...
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/10583#discussion_r48951642
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/CatalystQl.scala ---
@@ -0,0 +1,961 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst
+
+import java.sql.Date
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.analysis._
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.aggregate.Count
+import org.apache.spark.sql.catalyst.plans._
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.trees.CurrentOrigin
+import org.apache.spark.sql.catalyst.parser._
+import org.apache.spark.sql.types._
+import org.apache.spark.unsafe.types.CalendarInterval
+import org.apache.spark.util.random.RandomSampler
+
+/**
+ * This class translates a HQL String to a Catalyst [[LogicalPlan]] or
[[Expression]].
+ */
+private[sql] class CatalystQl(val conf: ParserConf = SimpleParserConf()) {
+ object Token {
+def unapply(node: ASTNode): Some[(String, List[ASTNode])] = {
+ CurrentOrigin.setPosition(node.line, node.positionInLine)
+ node.pattern
+}
+ }
+
+
+ /**
+ * Returns the AST for the given SQL string.
+ */
+ protected def getAst(sql: String): ASTNode = ParseDriver.parse(sql, conf)
+
+ /** Creates LogicalPlan for a given HiveQL string. */
+ def createPlan(sql: String): LogicalPlan = {
+try {
+ createPlan(sql, ParseDriver.parse(sql, conf))
+} catch {
+ case e: MatchError => throw e
+ case e: AnalysisException => throw e
+ case e: Exception =>
+throw new AnalysisException(e.getMessage)
+ case e: NotImplementedError =>
+throw new AnalysisException(
+ s"""
+ |Unsupported language features in query: $sql
+ |${getAst(sql).treeString}
+ |$e
+ |${e.getStackTrace.head}
+ """.stripMargin)
+}
+ }
+
+ protected def createPlan(sql: String, tree: ASTNode): LogicalPlan =
nodeToPlan(tree)
+
+ def parseDdl(ddl: String): Seq[Attribute] = {
+val tree = getAst(ddl)
+assert(tree.text == "TOK_CREATETABLE", "Only CREATE TABLE supported.")
+val tableOps = tree.children
+val colList = tableOps
+ .find(_.text == "TOK_TABCOLLIST")
+ .getOrElse(sys.error("No columnList!"))
+
+colList.children.map(nodeToAttribute)
+ }
+
+ protected def getClauses(
+ clauseNames: Seq[String],
+ nodeList: Seq[ASTNode]): Seq[Option[ASTNode]] = {
+var remainingNodes = nodeList
+val clauses = clauseNames.map { clauseName =>
+ val (matches, nonMatches) =
remainingNodes.partition(_.text.toUpperCase == clauseName)
+ remainingNodes = nonMatches ++ (if (matches.nonEmpty) matches.tail
else Nil)
+ matches.headOption
+}
+
+if (remainingNodes.nonEmpty) {
+ sys.error(
+s"""Unhandled clauses:
${remainingNodes.map(_.treeString).mkString("\n")}.
+|You are likely trying to use an unsupported Hive
feature..stripMargin)
+}
+clauses
+ }
+
+ protected def getClause(clauseName: String, nodeList: Seq[ASTNode]):
ASTNode =
+getClauseOption(clauseName, nodeList).getOrElse(sys.error(
+ s"Expected clause $clauseName missing from
${nodeList.map(_.treeString).mkString("\n")}"))
+
+ protected def getClauseOption(clauseName: String, nodeList:
Seq[ASTNode]): Option[ASTNode] = {
+nodeList.filter { case ast: ASTNode => ast.text == clauseName } match {
+ case Seq(oneMatch) => Some(oneMatch)
+ case Seq() => None
+ case _ => sys.error(s"Found multiple instances of clause
$clauseName")
+}
+ }
+
+
[GitHub] spark pull request: [SPARK-12573][SPARK-12574][SQL] Move SQL Parse...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10583#issuecomment-169326184 **[Test build #48853 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48853/consoleFull)** for PR 10583 at commit [`3680d4c`](https://github.com/apache/spark/commit/3680d4c4179d63584a42f6d33bbe3c718fa3075d). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12573][SPARK-12574][SQL] Move SQL Parse...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10583#issuecomment-169326444 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48853/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12573][SPARK-12574][SQL] Move SQL Parse...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10583#issuecomment-169326441 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12573][SPARK-12574][SQL] Move SQL Parse...
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/10583#issuecomment-169404802 Regarding the questions about `LATERAL VIEW`. Ideally I think we will support one query language that has a super-set of the features that were previously present in HiveQL and the SparkSQLParser. Where this is not possible I'd probably defer to HiveQL or a SQL Standard. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12573][SPARK-12574][SQL] Move SQL Parse...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/10583#discussion_r48988457
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/CatalystQl.scala ---
@@ -0,0 +1,961 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst
+
+import java.sql.Date
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.analysis._
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.aggregate.Count
+import org.apache.spark.sql.catalyst.plans._
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.trees.CurrentOrigin
+import org.apache.spark.sql.catalyst.parser._
+import org.apache.spark.sql.types._
+import org.apache.spark.unsafe.types.CalendarInterval
+import org.apache.spark.util.random.RandomSampler
+
+/**
+ * This class translates a HQL String to a Catalyst [[LogicalPlan]] or
[[Expression]].
+ */
+private[sql] class CatalystQl(val conf: ParserConf = SimpleParserConf()) {
+ object Token {
+def unapply(node: ASTNode): Some[(String, List[ASTNode])] = {
+ CurrentOrigin.setPosition(node.line, node.positionInLine)
+ node.pattern
+}
+ }
+
+
+ /**
+ * Returns the AST for the given SQL string.
+ */
+ protected def getAst(sql: String): ASTNode = ParseDriver.parse(sql, conf)
+
+ /** Creates LogicalPlan for a given HiveQL string. */
+ def createPlan(sql: String): LogicalPlan = {
+try {
+ createPlan(sql, ParseDriver.parse(sql, conf))
+} catch {
+ case e: MatchError => throw e
+ case e: AnalysisException => throw e
+ case e: Exception =>
+throw new AnalysisException(e.getMessage)
+ case e: NotImplementedError =>
+throw new AnalysisException(
+ s"""
+ |Unsupported language features in query: $sql
+ |${getAst(sql).treeString}
+ |$e
+ |${e.getStackTrace.head}
+ """.stripMargin)
+}
+ }
+
+ protected def createPlan(sql: String, tree: ASTNode): LogicalPlan =
nodeToPlan(tree)
+
+ def parseDdl(ddl: String): Seq[Attribute] = {
+val tree = getAst(ddl)
+assert(tree.text == "TOK_CREATETABLE", "Only CREATE TABLE supported.")
+val tableOps = tree.children
+val colList = tableOps
+ .find(_.text == "TOK_TABCOLLIST")
+ .getOrElse(sys.error("No columnList!"))
+
+colList.children.map(nodeToAttribute)
+ }
+
+ protected def getClauses(
+ clauseNames: Seq[String],
+ nodeList: Seq[ASTNode]): Seq[Option[ASTNode]] = {
+var remainingNodes = nodeList
+val clauses = clauseNames.map { clauseName =>
+ val (matches, nonMatches) =
remainingNodes.partition(_.text.toUpperCase == clauseName)
+ remainingNodes = nonMatches ++ (if (matches.nonEmpty) matches.tail
else Nil)
+ matches.headOption
+}
+
+if (remainingNodes.nonEmpty) {
+ sys.error(
+s"""Unhandled clauses:
${remainingNodes.map(_.treeString).mkString("\n")}.
+|You are likely trying to use an unsupported Hive
feature..stripMargin)
+}
+clauses
+ }
+
+ protected def getClause(clauseName: String, nodeList: Seq[ASTNode]):
ASTNode =
+getClauseOption(clauseName, nodeList).getOrElse(sys.error(
+ s"Expected clause $clauseName missing from
${nodeList.map(_.treeString).mkString("\n")}"))
+
+ protected def getClauseOption(clauseName: String, nodeList:
Seq[ASTNode]): Option[ASTNode] = {
+nodeList.filter { case ast: ASTNode => ast.text == clauseName } match {
+ case Seq(oneMatch) => Some(oneMatch)
+ case Seq() => None
+ case _ => sys.error(s"Found multiple instances of clause
$clauseName")
+}
+ }
+
+
[GitHub] spark pull request: [SPARK-12573][SPARK-12574][SQL] Move SQL Parse...
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/10583#discussion_r48995383
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/CatalystQl.scala ---
@@ -0,0 +1,961 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst
+
+import java.sql.Date
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.analysis._
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.aggregate.Count
+import org.apache.spark.sql.catalyst.plans._
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.trees.CurrentOrigin
+import org.apache.spark.sql.catalyst.parser._
+import org.apache.spark.sql.types._
+import org.apache.spark.unsafe.types.CalendarInterval
+import org.apache.spark.util.random.RandomSampler
+
+/**
+ * This class translates a HQL String to a Catalyst [[LogicalPlan]] or
[[Expression]].
+ */
+private[sql] class CatalystQl(val conf: ParserConf = SimpleParserConf()) {
+ object Token {
+def unapply(node: ASTNode): Some[(String, List[ASTNode])] = {
+ CurrentOrigin.setPosition(node.line, node.positionInLine)
+ node.pattern
+}
+ }
+
+
+ /**
+ * Returns the AST for the given SQL string.
+ */
+ protected def getAst(sql: String): ASTNode = ParseDriver.parse(sql, conf)
+
+ /** Creates LogicalPlan for a given HiveQL string. */
+ def createPlan(sql: String): LogicalPlan = {
+try {
+ createPlan(sql, ParseDriver.parse(sql, conf))
+} catch {
+ case e: MatchError => throw e
+ case e: AnalysisException => throw e
+ case e: Exception =>
+throw new AnalysisException(e.getMessage)
+ case e: NotImplementedError =>
+throw new AnalysisException(
+ s"""
+ |Unsupported language features in query: $sql
+ |${getAst(sql).treeString}
+ |$e
+ |${e.getStackTrace.head}
+ """.stripMargin)
+}
+ }
+
+ protected def createPlan(sql: String, tree: ASTNode): LogicalPlan =
nodeToPlan(tree)
+
+ def parseDdl(ddl: String): Seq[Attribute] = {
+val tree = getAst(ddl)
+assert(tree.text == "TOK_CREATETABLE", "Only CREATE TABLE supported.")
+val tableOps = tree.children
+val colList = tableOps
+ .find(_.text == "TOK_TABCOLLIST")
+ .getOrElse(sys.error("No columnList!"))
+
+colList.children.map(nodeToAttribute)
+ }
+
+ protected def getClauses(
+ clauseNames: Seq[String],
+ nodeList: Seq[ASTNode]): Seq[Option[ASTNode]] = {
+var remainingNodes = nodeList
+val clauses = clauseNames.map { clauseName =>
+ val (matches, nonMatches) =
remainingNodes.partition(_.text.toUpperCase == clauseName)
+ remainingNodes = nonMatches ++ (if (matches.nonEmpty) matches.tail
else Nil)
+ matches.headOption
+}
+
+if (remainingNodes.nonEmpty) {
+ sys.error(
+s"""Unhandled clauses:
${remainingNodes.map(_.treeString).mkString("\n")}.
+|You are likely trying to use an unsupported Hive
feature..stripMargin)
+}
+clauses
+ }
+
+ protected def getClause(clauseName: String, nodeList: Seq[ASTNode]):
ASTNode =
+getClauseOption(clauseName, nodeList).getOrElse(sys.error(
+ s"Expected clause $clauseName missing from
${nodeList.map(_.treeString).mkString("\n")}"))
+
+ protected def getClauseOption(clauseName: String, nodeList:
Seq[ASTNode]): Option[ASTNode] = {
+nodeList.filter { case ast: ASTNode => ast.text == clauseName } match {
+ case Seq(oneMatch) => Some(oneMatch)
+ case Seq() => None
+ case _ => sys.error(s"Found multiple instances of clause
$clauseName")
+}
+ }
+
+
[GitHub] spark pull request: [SPARK-12573][SPARK-12574][SQL] Move SQL Parse...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/10583#issuecomment-169425486 Due to the size of the patch, I'm going to merge this in now. @hvanhovell can address more comments as follow-up prs. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12573][SPARK-12574][SQL] Move SQL Parse...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/10583 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12573][SPARK-12574][SQL] Move SQL Parse...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10583#issuecomment-168932606 **[Test build #48747 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48747/consoleFull)** for PR 10583 at commit [`fb3b4a4`](https://github.com/apache/spark/commit/fb3b4a4c461391866bc12a51dd1e60eadeaff916). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12573][SPARK-12574][SQL] Move SQL Parse...
Github user hvanhovell commented on the pull request: https://github.com/apache/spark/pull/10583#issuecomment-168929936 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12573][SPARK-12574][SQL] Move SQL Parse...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10583#issuecomment-168931844 **[Test build #2322 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2322/consoleFull)** for PR 10583 at commit [`fb3b4a4`](https://github.com/apache/spark/commit/fb3b4a4c461391866bc12a51dd1e60eadeaff916). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12573][SPARK-12574][SQL] Move SQL Parse...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10583#issuecomment-168955204 **[Test build #2322 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2322/consoleFull)** for PR 10583 at commit [`fb3b4a4`](https://github.com/apache/spark/commit/fb3b4a4c461391866bc12a51dd1e60eadeaff916). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12573][SPARK-12574][SQL] Move SQL Parse...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10583#issuecomment-168965051 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48747/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12573][SPARK-12574][SQL] Move SQL Parse...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10583#issuecomment-168965050 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12573][SPARK-12574][SQL] Move SQL Parse...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10583#issuecomment-168964867 **[Test build #48747 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48747/consoleFull)** for PR 10583 at commit [`fb3b4a4`](https://github.com/apache/spark/commit/fb3b4a4c461391866bc12a51dd1e60eadeaff916). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12573][SPARK-12574][SQL] Move SQL Parse...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10583#issuecomment-169049675 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48771/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12573][SPARK-12574][SQL] Move SQL Parse...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10583#issuecomment-169049668 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12573][SPARK-12574][SQL] Move SQL Parse...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10583#issuecomment-169049383 **[Test build #48771 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48771/consoleFull)** for PR 10583 at commit [`43c29b7`](https://github.com/apache/spark/commit/43c29b7a2ba3598e50561a974dba1d763e90746c). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12573][SPARK-12574][SQL] Move SQL Parse...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10583#issuecomment-169007832 **[Test build #48766 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48766/consoleFull)** for PR 10583 at commit [`e397370`](https://github.com/apache/spark/commit/e39737023920c3916ad8ed6e4d4b46072bfe4f7a). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12573][SPARK-12574][SQL] Move SQL Parse...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10583#issuecomment-169007990 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12573][SPARK-12574][SQL] Move SQL Parse...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10583#issuecomment-169007994 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48766/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12573][SPARK-12574][SQL] Move SQL Parse...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10583#issuecomment-169012694 **[Test build #48771 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48771/consoleFull)** for PR 10583 at commit [`43c29b7`](https://github.com/apache/spark/commit/43c29b7a2ba3598e50561a974dba1d763e90746c). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12573][SPARK-12574][SQL] Move SQL Parse...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10583#issuecomment-169094038 **[Test build #48780 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48780/consoleFull)** for PR 10583 at commit [`157d178`](https://github.com/apache/spark/commit/157d1785a8362f17700f18106ec4aba5d70dc90f). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12573][SPARK-12574][SQL] Move SQL Parse...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10583#issuecomment-169122642 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48780/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12573][SPARK-12574][SQL] Move SQL Parse...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10583#issuecomment-169122639 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12573][SPARK-12574][SQL] Move SQL Parse...
Github user hvanhovell commented on the pull request: https://github.com/apache/spark/pull/10583#issuecomment-169126939 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12573][SPARK-12574][SQL] Move SQL Parse...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10583#issuecomment-169129174 **[Test build #48789 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48789/consoleFull)** for PR 10583 at commit [`157d178`](https://github.com/apache/spark/commit/157d1785a8362f17700f18106ec4aba5d70dc90f). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12573][SPARK-12574][SQL] Move SQL Parse...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10583#issuecomment-169122427 **[Test build #48780 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48780/consoleFull)** for PR 10583 at commit [`157d178`](https://github.com/apache/spark/commit/157d1785a8362f17700f18106ec4aba5d70dc90f). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12573][SPARK-12574][SQL] Move SQL Parse...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/10583#issuecomment-169208476 BTW given the size of the pull request, I think we can also merge it provided that it has no structural problems, and then review feedback in follow-up prs. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12573][SPARK-12574][SQL] Move SQL Parse...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/10583#discussion_r48922609
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLConf.scala ---
@@ -451,6 +452,19 @@ private[spark] object SQLConf {
doc = "When true, we could use `datasource`.`path` as table in SQL
query"
)
+ val PARSER_SUPPORT_QUOTEDID =
stringConf("spark.sql.parser.supportQuotedIdentifiers",
--- End diff --
should this be a boolean conf?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12573][SPARK-12574][SQL] Move SQL Parse...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/10583#discussion_r48923332
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/CatalystQl.scala ---
@@ -0,0 +1,969 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst
+
+import java.sql.Date
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.analysis._
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.aggregate.Count
+import org.apache.spark.sql.catalyst.plans._
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.trees.CurrentOrigin
+import org.apache.spark.sql.catalyst.parser._
+import org.apache.spark.sql.types._
+import org.apache.spark.unsafe.types.CalendarInterval
+import org.apache.spark.util.random.RandomSampler
+
+/**
+ * This class translates a HQL String to a Catalyst [[LogicalPlan]] or
[[Expression]].
+ */
+private[sql] class CatalystQl(val conf: ParserConf = SimpleParserConf()) {
+ object Token {
+def unapply(node: ASTNode): Some[(String, List[ASTNode])] = {
+ CurrentOrigin.setPosition(node.line, node.positionInLine)
+ node.pattern
+}
+ }
+
+ // TODO improve the parse error - so we don't need this anymore.
+ val errorRegEx = "line (\\d+):(\\d+) (.*)".r
+
+ /**
+ * Returns the AST for the given SQL string.
+ */
+ protected def getAst(sql: String): ASTNode = ParseDriver.parse(sql, conf)
+
+ /** Creates LogicalPlan for a given HiveQL string. */
+ def createPlan(sql: String): LogicalPlan = {
+try {
+ createPlan(sql, ParseDriver.parse(sql, conf))
+} catch {
+ case pe: ParseException =>
+pe.getMessage match {
+ case errorRegEx(line, start, message) =>
+throw new AnalysisException(message, Some(line.toInt),
Some(start.toInt))
+ case otherMessage =>
+throw new AnalysisException(otherMessage)
+}
+ case e: MatchError => throw e
+ case e: Exception =>
+throw new AnalysisException(e.getMessage)
+ case e: NotImplementedError =>
+throw new AnalysisException(
+ s"""
+ |Unsupported language features in query: $sql
+ |${getAst(sql).treeString}
+ |$e
+ |${e.getStackTrace.head}
+ """.stripMargin)
+}
+ }
+
+ protected def createPlan(sql: String, tree: ASTNode): LogicalPlan =
nodeToPlan(tree)
+
+ def parseDdl(ddl: String): Seq[Attribute] = {
+val tree =
+ try {
+getAst(ddl)
+ } catch {
+case pe: ParseException =>
+ throw new RuntimeException(s"Failed to parse ddl: '$ddl'", pe)
+ }
+assert(tree.text == "TOK_CREATETABLE", "Only CREATE TABLE supported.")
+val tableOps = tree.children
+val colList = tableOps
+ .find(_.text == "TOK_TABCOLLIST")
+ .getOrElse(sys.error("No columnList!"))
+
+colList.children.map(nodeToAttribute)
+ }
+
+ protected def getClauses(
+ clauseNames: Seq[String],
+ nodeList: Seq[ASTNode]): Seq[Option[ASTNode]] = {
+var remainingNodes = nodeList
+val clauses = clauseNames.map { clauseName =>
+ val (matches, nonMatches) =
remainingNodes.partition(_.text.toUpperCase == clauseName)
+ remainingNodes = nonMatches ++ (if (matches.nonEmpty) matches.tail
else Nil)
+ matches.headOption
+}
+
+if (remainingNodes.nonEmpty) {
+ sys.error(
+s"""Unhandled clauses:
${remainingNodes.map(_.treeString).mkString("\n")}.
+|You are likely trying to use an unsupported Hive
feature..stripMargin)
+}
+clauses
+ }
+
+ protected def getClause(clauseName: String,
[GitHub] spark pull request: [SPARK-12573][SPARK-12574][SQL] Move SQL Parse...
Github user cloud-fan commented on the pull request: https://github.com/apache/spark/pull/10583#issuecomment-169204037 Does case-sensitivity config still work with the new parser? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12573][SPARK-12574][SQL] Move SQL Parse...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/10583#discussion_r48923378
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/CatalystQl.scala ---
@@ -0,0 +1,969 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst
+
+import java.sql.Date
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.analysis._
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.aggregate.Count
+import org.apache.spark.sql.catalyst.plans._
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.trees.CurrentOrigin
+import org.apache.spark.sql.catalyst.parser._
+import org.apache.spark.sql.types._
+import org.apache.spark.unsafe.types.CalendarInterval
+import org.apache.spark.util.random.RandomSampler
+
+/**
+ * This class translates a HQL String to a Catalyst [[LogicalPlan]] or
[[Expression]].
+ */
+private[sql] class CatalystQl(val conf: ParserConf = SimpleParserConf()) {
+ object Token {
+def unapply(node: ASTNode): Some[(String, List[ASTNode])] = {
+ CurrentOrigin.setPosition(node.line, node.positionInLine)
+ node.pattern
+}
+ }
+
+ // TODO improve the parse error - so we don't need this anymore.
+ val errorRegEx = "line (\\d+):(\\d+) (.*)".r
+
+ /**
+ * Returns the AST for the given SQL string.
+ */
+ protected def getAst(sql: String): ASTNode = ParseDriver.parse(sql, conf)
+
+ /** Creates LogicalPlan for a given HiveQL string. */
+ def createPlan(sql: String): LogicalPlan = {
+try {
+ createPlan(sql, ParseDriver.parse(sql, conf))
+} catch {
+ case pe: ParseException =>
+pe.getMessage match {
+ case errorRegEx(line, start, message) =>
+throw new AnalysisException(message, Some(line.toInt),
Some(start.toInt))
+ case otherMessage =>
+throw new AnalysisException(otherMessage)
+}
+ case e: MatchError => throw e
+ case e: Exception =>
+throw new AnalysisException(e.getMessage)
+ case e: NotImplementedError =>
+throw new AnalysisException(
+ s"""
+ |Unsupported language features in query: $sql
+ |${getAst(sql).treeString}
+ |$e
+ |${e.getStackTrace.head}
+ """.stripMargin)
+}
+ }
+
+ protected def createPlan(sql: String, tree: ASTNode): LogicalPlan =
nodeToPlan(tree)
+
+ def parseDdl(ddl: String): Seq[Attribute] = {
+val tree =
+ try {
+getAst(ddl)
+ } catch {
+case pe: ParseException =>
+ throw new RuntimeException(s"Failed to parse ddl: '$ddl'", pe)
+ }
+assert(tree.text == "TOK_CREATETABLE", "Only CREATE TABLE supported.")
+val tableOps = tree.children
+val colList = tableOps
+ .find(_.text == "TOK_TABCOLLIST")
+ .getOrElse(sys.error("No columnList!"))
+
+colList.children.map(nodeToAttribute)
+ }
+
+ protected def getClauses(
+ clauseNames: Seq[String],
+ nodeList: Seq[ASTNode]): Seq[Option[ASTNode]] = {
+var remainingNodes = nodeList
+val clauses = clauseNames.map { clauseName =>
+ val (matches, nonMatches) =
remainingNodes.partition(_.text.toUpperCase == clauseName)
+ remainingNodes = nonMatches ++ (if (matches.nonEmpty) matches.tail
else Nil)
+ matches.headOption
+}
+
+if (remainingNodes.nonEmpty) {
+ sys.error(
+s"""Unhandled clauses:
${remainingNodes.map(_.treeString).mkString("\n")}.
+|You are likely trying to use an unsupported Hive
feature..stripMargin)
+}
+clauses
+ }
+
+ protected def getClause(clauseName: String,
[GitHub] spark pull request: [SPARK-12573][SPARK-12574][SQL] Move SQL Parse...
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/10583#discussion_r48934251
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/CatalystQl.scala ---
@@ -0,0 +1,969 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst
+
+import java.sql.Date
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.analysis._
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.aggregate.Count
+import org.apache.spark.sql.catalyst.plans._
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.trees.CurrentOrigin
+import org.apache.spark.sql.catalyst.parser._
+import org.apache.spark.sql.types._
+import org.apache.spark.unsafe.types.CalendarInterval
+import org.apache.spark.util.random.RandomSampler
+
+/**
+ * This class translates a HQL String to a Catalyst [[LogicalPlan]] or
[[Expression]].
+ */
+private[sql] class CatalystQl(val conf: ParserConf = SimpleParserConf()) {
+ object Token {
+def unapply(node: ASTNode): Some[(String, List[ASTNode])] = {
+ CurrentOrigin.setPosition(node.line, node.positionInLine)
+ node.pattern
+}
+ }
+
+ // TODO improve the parse error - so we don't need this anymore.
+ val errorRegEx = "line (\\d+):(\\d+) (.*)".r
+
+ /**
+ * Returns the AST for the given SQL string.
+ */
+ protected def getAst(sql: String): ASTNode = ParseDriver.parse(sql, conf)
+
+ /** Creates LogicalPlan for a given HiveQL string. */
+ def createPlan(sql: String): LogicalPlan = {
+try {
+ createPlan(sql, ParseDriver.parse(sql, conf))
+} catch {
+ case pe: ParseException =>
+pe.getMessage match {
+ case errorRegEx(line, start, message) =>
+throw new AnalysisException(message, Some(line.toInt),
Some(start.toInt))
+ case otherMessage =>
+throw new AnalysisException(otherMessage)
+}
+ case e: MatchError => throw e
+ case e: Exception =>
+throw new AnalysisException(e.getMessage)
+ case e: NotImplementedError =>
+throw new AnalysisException(
+ s"""
+ |Unsupported language features in query: $sql
+ |${getAst(sql).treeString}
+ |$e
+ |${e.getStackTrace.head}
+ """.stripMargin)
+}
+ }
+
+ protected def createPlan(sql: String, tree: ASTNode): LogicalPlan =
nodeToPlan(tree)
+
+ def parseDdl(ddl: String): Seq[Attribute] = {
+val tree =
+ try {
+getAst(ddl)
+ } catch {
+case pe: ParseException =>
+ throw new RuntimeException(s"Failed to parse ddl: '$ddl'", pe)
+ }
+assert(tree.text == "TOK_CREATETABLE", "Only CREATE TABLE supported.")
+val tableOps = tree.children
+val colList = tableOps
+ .find(_.text == "TOK_TABCOLLIST")
+ .getOrElse(sys.error("No columnList!"))
+
+colList.children.map(nodeToAttribute)
+ }
+
+ protected def getClauses(
+ clauseNames: Seq[String],
+ nodeList: Seq[ASTNode]): Seq[Option[ASTNode]] = {
+var remainingNodes = nodeList
+val clauses = clauseNames.map { clauseName =>
+ val (matches, nonMatches) =
remainingNodes.partition(_.text.toUpperCase == clauseName)
+ remainingNodes = nonMatches ++ (if (matches.nonEmpty) matches.tail
else Nil)
+ matches.headOption
+}
+
+if (remainingNodes.nonEmpty) {
+ sys.error(
+s"""Unhandled clauses:
${remainingNodes.map(_.treeString).mkString("\n")}.
+|You are likely trying to use an unsupported Hive
feature..stripMargin)
+}
+clauses
+ }
+
+ protected def getClause(clauseName: String,
[GitHub] spark pull request: [SPARK-12573][SPARK-12574][SQL] Move SQL Parse...
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/10583#discussion_r48934317
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLConf.scala ---
@@ -451,6 +452,19 @@ private[spark] object SQLConf {
doc = "When true, we could use `datasource`.`path` as table in SQL
query"
)
+ val PARSER_SUPPORT_QUOTEDID =
stringConf("spark.sql.parser.supportQuotedIdentifiers",
--- End diff --
I ported this directly form Hive. It has only two options:
- ```none```: no quoting. I'll map this to false.
- ```column```: quoting. I'll map this to true.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12573][SPARK-12574][SQL] Move SQL Parse...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/10583#discussion_r48919353
--- Diff:
sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/parser/ParseUtils.java
---
@@ -0,0 +1,163 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.parser;
+
+import java.io.UnsupportedEncodingException;
+
+/**
+ * A couple of utility methods that help with parsing ASTs.
+ *
+ * Both methods in this class were take from the SemanticAnalyzer in Hive:
+ * ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
+ */
+public final class ParseUtils {
+ private ParseUtils() {
+super();
+ }
+
+ public static String charSetString(String charSetName, String
charSetString)
+ throws UnsupportedEncodingException {
+ // The character set name starts with a _, so strip that
--- End diff --
looks like the indentation is off here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12573][SPARK-12574][SQL] Move SQL Parse...
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/10583#discussion_r48919450 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -587,6 +586,13 @@ class Analyzer( case other => other } } + case u @ UnresolvedGenerator(name, children) => --- End diff -- cc @yhuai for this change --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12573][SPARK-12574][SQL] Move SQL Parse...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/10583#discussion_r48919389
--- Diff:
sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/parser/ParseUtils.java
---
@@ -0,0 +1,163 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.parser;
+
+import java.io.UnsupportedEncodingException;
+
+/**
+ * A couple of utility methods that help with parsing ASTs.
+ *
+ * Both methods in this class were take from the SemanticAnalyzer in Hive:
+ * ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
+ */
+public final class ParseUtils {
+ private ParseUtils() {
+super();
+ }
+
+ public static String charSetString(String charSetName, String
charSetString)
--- End diff --
are there unit tests from Hive that we can take?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12573][SPARK-12574][SQL] Move SQL Parse...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/10583#discussion_r48919495
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/unresolved.scala
---
@@ -129,23 +129,39 @@ object UnresolvedAttribute {
nameParts.toSeq
}
}
+trait UnresolvedFunctionLike extends Expression {
--- End diff --
I'd remove this trait since it is only used twice. The overhead of having
an abstraction doesn't seem worth it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12573][SPARK-12574][SQL] Move SQL Parse...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10583#issuecomment-169165991 **[Test build #48789 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48789/consoleFull)** for PR 10583 at commit [`157d178`](https://github.com/apache/spark/commit/157d1785a8362f17700f18106ec4aba5d70dc90f). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12573][SPARK-12574][SQL] Move SQL Parse...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10583#issuecomment-169166339 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48789/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12573][SPARK-12574][SQL] Move SQL Parse...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10583#issuecomment-169166337 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12573][SPARK-12574][SQL] Move SQL Parse...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/10583#issuecomment-169169980 Can you update the pull request description? It still says WIP. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12573][SPARK-12574][SQL] Move SQL Parse...
Github user hvanhovell commented on the pull request: https://github.com/apache/spark/pull/10583#issuecomment-169171058 Done. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12573][SPARK-12574][SQL] Move SQL Parse...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10583#issuecomment-168984129 **[Test build #48766 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48766/consoleFull)** for PR 10583 at commit [`e397370`](https://github.com/apache/spark/commit/e39737023920c3916ad8ed6e4d4b46072bfe4f7a). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12573][SPARK-12574][SQL] Move SQL Parse...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/10583#issuecomment-169196055 cc @cloud-fan can you take a look at this? Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12573][SPARK-12574][SQL] Move SQL Parse...
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/10583#discussion_r48921238 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -587,6 +586,13 @@ class Analyzer( case other => other } } + case u @ UnresolvedGenerator(name, children) => --- End diff -- Do we need to add `UnresolvedGenerator`? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12573][SPARK-12574][SQL] Move SQL Parse...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10583#issuecomment-168866570 **[Test build #48709 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48709/consoleFull)** for PR 10583 at commit [`fb3b4a4`](https://github.com/apache/spark/commit/fb3b4a4c461391866bc12a51dd1e60eadeaff916). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12573][SPARK-12574][SQL] Move SQL Parse...
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/10583#discussion_r48805244 --- Diff: sql/catalyst/src/main/java/org/apache/spark/sql/parser/ParseUtils.java --- @@ -0,0 +1,163 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.spark.sql.parser; --- End diff -- Yeah (reynold just suggested the same thing), I'll add it to catalyst.parser. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12573][SPARK-12574][SQL] Move SQL Parse...
Github user marmbrus commented on a diff in the pull request: https://github.com/apache/spark/pull/10583#discussion_r48805084 --- Diff: sql/catalyst/src/main/java/org/apache/spark/sql/parser/ParseUtils.java --- @@ -0,0 +1,163 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.spark.sql.parser; --- End diff -- Should we put this in `catalyst`? Just since we tend to hide non-public APIs there. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12573][SPARK-12574][SQL] Move SQL Parse...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10583#issuecomment-168906958 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48709/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12573][SPARK-12574][SQL] Move SQL Parse...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10583#issuecomment-168906918 **[Test build #48709 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48709/consoleFull)** for PR 10583 at commit [`fb3b4a4`](https://github.com/apache/spark/commit/fb3b4a4c461391866bc12a51dd1e60eadeaff916). * This patch **fails from timeout after a configured wait of \`250m\`**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12573][SPARK-12574][SQL] Move SQL Parse...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10583#issuecomment-168906957 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
