[GitHub] spark pull request: [SPARK-14868][BUILD] Enable NewLineAtEofChecke...
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/12632#discussion_r60839977
--- Diff:
mllib/src/test/java/org/apache/spark/ml/classification/JavaRandomForestClassifierSuite.java
---
@@ -81,15 +81,15 @@ public void runDT() {
for (String featureSubsetStrategy:
RandomForestClassifier.supportedFeatureSubsetStrategies()) {
rf.setFeatureSubsetStrategy(featureSubsetStrategy);
}
-String realStrategies[] = {".1", ".10", "0.10", "0.1", "0.9", "1.0"};
+String[] realStrategies = {".1", ".10", "0.10", "0.1", "0.9", "1.0"};
--- End diff --
Sure. That's `ArrayTypeStyle` rule.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14867][BUILD] Remove `--force` option i...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12631#issuecomment-213905832 **[Test build #56836 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56836/consoleFull)** for PR 12631 at commit [`b11de8a`](https://github.com/apache/spark/commit/b11de8af931d0390d4d2e89aba61791410fe6984). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14865][SQL] Better error handling for v...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/12633#discussion_r60839961
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/views.scala ---
@@ -85,68 +88,74 @@ case class CreateViewCommand(
} else {
// Create the view if it doesn't exist.
sessionState.catalog.createTable(
-prepareTable(sqlContext, analzyedPlan), ignoreIfExists = false)
+prepareTable(sqlContext, analyzedPlan), ignoreIfExists = false)
}
Seq.empty[Row]
}
- private def prepareTable(sqlContext: SQLContext, analzyedPlan:
LogicalPlan): CatalogTable = {
-val expandedText = if (sqlContext.conf.canonicalView) {
- try rebuildViewQueryString(sqlContext, analzyedPlan) catch {
-case NonFatal(e) => wrapViewTextWithSelect(analzyedPlan)
+ /**
+ * Returns a [[CatalogTable]] that can be used to save in the catalog.
This comment canonicalize
+ * SQL based on the analyzed plan, and also creates the proper schema
for the view.
+ */
+ private def prepareTable(sqlContext: SQLContext, analyzedPlan:
LogicalPlan): CatalogTable = {
+val viewSQL: String =
+ if (sqlContext.conf.canonicalView) {
+val logicalPlan =
+ if (tableDesc.schema.isEmpty) {
+analyzedPlan
+ } else {
+val projectList =
analyzedPlan.output.zip(tableDesc.schema).map {
+ case (attr, col) => Alias(attr, col.name)()
+}
+sqlContext.executePlan(Project(projectList,
analyzedPlan)).analyzed
+ }
+new SQLBuilder(logicalPlan).toSQL
+ } else {
+// When user specified column names for view, we should create a
project to do the renaming.
+// When no column name specified, we still need to create a
project to declare the columns
+// we need, to make us more robust to top level `*`s.
+val viewOutput = {
+ val columnNames = analyzedPlan.output.map(f => quote(f.name))
+ if (tableDesc.schema.isEmpty) {
+columnNames.mkString(", ")
+ } else {
+columnNames.zip(tableDesc.schema.map(f => quote(f.name))).map {
+ case (name, alias) => s"$name AS $alias"
+}.mkString(", ")
+ }
+}
+
+val viewText = tableDesc.viewText.get
+val viewName = quote(tableDesc.identifier.table)
+s"SELECT $viewOutput FROM ($viewText) $viewName"
}
-} else {
- wrapViewTextWithSelect(analzyedPlan)
+
+// Validate the view SQL - make sure we can parse it and analyze it.
+// If we cannot analyze the generated query, there is probably a bug
in SQL generation.
+try {
+ sqlContext.sql(viewSQL).queryExecution.assertAnalyzed()
+} catch {
+ case NonFatal(e) =>
+throw new RuntimeException(
+ "Failed to analyze the canonicalized SQL. It is possible there
is a bug in Spark.", e)
}
-val viewSchema = {
+val viewSchema: Seq[CatalogColumn] = {
if (tableDesc.schema.isEmpty) {
-analzyedPlan.output.map { a =>
+analyzedPlan.output.map { a =>
CatalogColumn(a.name, a.dataType.simpleString)
}
} else {
-analzyedPlan.output.zip(tableDesc.schema).map { case (a, col) =>
+analyzedPlan.output.zip(tableDesc.schema).map { case (a, col) =>
CatalogColumn(col.name, a.dataType.simpleString, nullable =
true, col.comment)
}
}
}
-tableDesc.copy(schema = viewSchema, viewText = Some(expandedText))
- }
-
- private def wrapViewTextWithSelect(analzyedPlan: LogicalPlan): String = {
-// When user specified column names for view, we should create a
project to do the renaming.
-// When no column name specified, we still need to create a project to
declare the columns
-// we need, to make us more robust to top level `*`s.
-val viewOutput = {
- val columnNames = analzyedPlan.output.map(f => quote(f.name))
- if (tableDesc.schema.isEmpty) {
-columnNames.mkString(", ")
- } else {
-columnNames.zip(tableDesc.schema.map(f => quote(f.name))).map {
- case (name, alias) => s"$name AS $alias"
-}.mkString(", ")
- }
-}
-
-val viewText = tableDesc.viewText.get
-val viewName = quote(tableDesc.identifier.table)
-s"SELECT $viewOutput FROM ($viewText) $viewName"
- }
-
- private def rebuildViewQueryString(sqlContext: SQLContext, analzyedPlan:
LogicalPlan): String = {
-val logicalPlan = if (tableDesc.schema.i
[GitHub] spark pull request: [SPARK-14883][DOCS] Fix wrong R examples and m...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12649#issuecomment-213905769 **[Test build #56835 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56835/consoleFull)** for PR 12649 at commit [`5d6d45e`](https://github.com/apache/spark/commit/5d6d45e07c15d17c5d1972733962013a6fcd228c). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14876][SQL] SparkSession should be case...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12643#issuecomment-213905767 **[Test build #56834 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56834/consoleFull)** for PR 12643 at commit [`fd95cb7`](https://github.com/apache/spark/commit/fd95cb7ef6e9ed331abe07d9a6cf15a73f3062b4). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14868][BUILD] Enable NewLineAtEofChecke...
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/12632#discussion_r60839935
--- Diff:
core/src/main/java/org/apache/spark/util/collection/unsafe/sort/PrefixComparators.java
---
@@ -82,37 +81,37 @@ public static long computePrefix(double value) {
//
public static final class UnsignedPrefixComparator extends
RadixSortSupport {
-@Override public final boolean sortDescending() { return false; }
-@Override public final boolean sortSigned() { return false; }
+@Override public boolean sortDescending() { return false; }
+@Override public boolean sortSigned() { return false; }
@Override
-public final int compare(long aPrefix, long bPrefix) {
+public int compare(long aPrefix, long bPrefix) {
return UnsignedLongs.compare(aPrefix, bPrefix);
}
}
public static final class UnsignedPrefixComparatorDesc extends
RadixSortSupport {
-@Override public final boolean sortDescending() { return true; }
-@Override public final boolean sortSigned() { return false; }
+@Override public boolean sortDescending() { return true; }
+@Override public boolean sortSigned() { return false; }
@Override
-public final int compare(long bPrefix, long aPrefix) {
+public int compare(long bPrefix, long aPrefix) {
--- End diff --
Oh, it's definitely final. It's just `RedundantModifier` error since the
class `SignedPrefixComparator` is already `final`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14883][DOCS] Fix wrong R examples and m...
GitHub user dongjoon-hyun opened a pull request:
https://github.com/apache/spark/pull/12649
[SPARK-14883][DOCS] Fix wrong R examples and make them up-to-date
## What changes were proposed in this pull request?
This issue aims to fix some errors in R examples and make them up-to-date
in docs and example modules.
- Fix the wrong usage of map. We need to use `lapply` if needed. However,
the usage of `lapply` also needs to be reviewed since it's private.
```
-teenNames <- map(teenagers, function(p) { paste("Name:", p$name)})
+teenNames <- SparkR:::lapply(teenagers, function(p) { paste("Name:",
p$name) })
```
- Fix the wrong example in Section `Generic Load/Save Functions` of
`docs/sql-programming-guide.md` for consistency
- Fix datatypes in `sparkr.md`.
- Update a data result in `sparkr.md`.
- Replace deprecated functions to remove warnings: jsonFile -> read.json,
parquetFile -> read.parquet
- Use up-to-date R-like functions: loadDF -> read.df, saveDF -> write.df,
saveAsParquetFile -> write.parquet
- Replace `SparkR DataFrame` with `SparkDataFrame` in `dataframe.R` and
`data-manipulation.R`.
- Other minor syntax fixes and a typo.
## How was this patch tested?
Manual.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dongjoon-hyun/spark SPARK-14883
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/12649.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #12649
commit 5d6d45e07c15d17c5d1972733962013a6fcd228c
Author: Dongjoon Hyun
Date: 2016-04-24T06:43:45Z
[SPARK-14883][DOCS] Fix wrong R examples and make them up-to-date
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14721][SQL] Remove HiveContext (part 2)
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/12585#issuecomment-213905492 The changes look good to me. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14868][BUILD] Enable NewLineAtEofChecke...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/12632#discussion_r60839755
--- Diff:
core/src/main/java/org/apache/spark/util/collection/unsafe/sort/PrefixComparators.java
---
@@ -82,37 +81,37 @@ public static long computePrefix(double value) {
//
public static final class UnsignedPrefixComparator extends
RadixSortSupport {
-@Override public final boolean sortDescending() { return false; }
-@Override public final boolean sortSigned() { return false; }
+@Override public boolean sortDescending() { return false; }
+@Override public boolean sortSigned() { return false; }
@Override
-public final int compare(long aPrefix, long bPrefix) {
+public int compare(long aPrefix, long bPrefix) {
return UnsignedLongs.compare(aPrefix, bPrefix);
}
}
public static final class UnsignedPrefixComparatorDesc extends
RadixSortSupport {
-@Override public final boolean sortDescending() { return true; }
-@Override public final boolean sortSigned() { return false; }
+@Override public boolean sortDescending() { return true; }
+@Override public boolean sortSigned() { return false; }
@Override
-public final int compare(long bPrefix, long aPrefix) {
+public int compare(long bPrefix, long aPrefix) {
--- End diff --
why can't this be final?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14868][BUILD] Enable NewLineAtEofChecke...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/12632#discussion_r60839760
--- Diff:
mllib/src/test/java/org/apache/spark/ml/classification/JavaRandomForestClassifierSuite.java
---
@@ -81,15 +81,15 @@ public void runDT() {
for (String featureSubsetStrategy:
RandomForestClassifier.supportedFeatureSubsetStrategies()) {
rf.setFeatureSubsetStrategy(featureSubsetStrategy);
}
-String realStrategies[] = {".1", ".10", "0.10", "0.1", "0.9", "1.0"};
+String[] realStrategies = {".1", ".10", "0.10", "0.1", "0.9", "1.0"};
--- End diff --
is there a rule for this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14487][SQL] User Defined Type registrat...
Github user viirya commented on the pull request: https://github.com/apache/spark/pull/12259#issuecomment-213897223 @mengxr @dbtsai Your comments are addressed now. Please take a look again when you have a change. Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14876][SQL] SparkSession should be case...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12643#issuecomment-213896260 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14876][SQL] SparkSession should be case...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12643#issuecomment-213896261 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/56833/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14876][SQL] SparkSession should be case...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12643#issuecomment-213896243 **[Test build #56833 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56833/consoleFull)** for PR 12643 at commit [`40aefc0`](https://github.com/apache/spark/commit/40aefc081dcd11f610e2199d11c2002236280153). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14867][BUILD] Remove `--force` option i...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12631#issuecomment-213892900 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14867][BUILD] Remove `--force` option i...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12631#issuecomment-213892901 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/56830/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14867][BUILD] Remove `--force` option i...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12631#issuecomment-213892810 **[Test build #56830 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56830/consoleFull)** for PR 12631 at commit [`8504cce`](https://github.com/apache/spark/commit/8504cce65231c38e70278c4c625aa3c87cb4c0c3). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14548][SQL] Support not greater than an...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/12316#issuecomment-213892774 @hvanhovell Can you think of any downsides if we support this? It is a little bit esoteric. The main one I can think of is whether this would set a precedent in that a lot of people start to submit "compatibility" things for esoteric features for different databases. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14881][PYTHON][SPARKR] pyspark and spar...
Github user felixcheung commented on the pull request: https://github.com/apache/spark/pull/12648#issuecomment-213892725 Hmmm... ``` [info] Run completed in 1 hour, 18 minutes, 30 seconds. [info] Total number of tests run: 1805 [info] Suites: completed 65, aborted 0 [info] Tests: succeeded 1805, failed 0, canceled 0, ignored 594, pending 0 [info] All tests passed. [info] Passed: Total 1810, Failed 0, Errors 0, Passed 1810, Ignored 594 [error] (core/test:test) sbt.TestsFailedException: Tests unsuccessful [error] Total time: 4879 s, completed Apr 23, 2016 6:42:17 PM [error] running /home/jenkins/workspace/SparkPullRequestBuilder/build/sbt -Pyarn -Phadoop-2.3 -Phive -Pkinesis-asl -Phive-thriftserver -Dtest.exclude.tags=org.apache.spark.tags.ExtendedHiveTest,org.apache.spark.tags.ExtendedYarnTest test ; received return code 1 ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Support single argument version of sqlContext....
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/12488 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14867][BUILD] Remove `--force` option i...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12631#issuecomment-213892530 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/56829/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14867][BUILD] Remove `--force` option i...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12631#issuecomment-213892529 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Support single argument version of sqlContext....
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/12488#issuecomment-213892519 hmm is there a JIRA ticket associated with this that you have created? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Support single argument version of sqlContext....
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/12488#issuecomment-213892503 Thanks - merging in master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14867][BUILD] Remove `--force` option i...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12631#issuecomment-213892482 **[Test build #56829 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56829/consoleFull)** for PR 12631 at commit [`cfe9b4e`](https://github.com/apache/spark/commit/cfe9b4eb101b9497a76938967b931ba99bbe6c9f). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14487][SQL] User Defined Type registrat...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12259#issuecomment-213892427 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14487][SQL] User Defined Type registrat...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12259#issuecomment-213892428 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/56831/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14487][SQL] User Defined Type registrat...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12259#issuecomment-213892392 **[Test build #56831 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56831/consoleFull)** for PR 12259 at commit [`06bdbc5`](https://github.com/apache/spark/commit/06bdbc518e7e85ce8627028c628d93f210f07cd8). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14879] [SQL] Move CreateMetastoreDataSo...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/12645 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14879] [SQL] Move CreateMetastoreDataSo...
Github user yhuai commented on the pull request: https://github.com/apache/spark/pull/12645#issuecomment-213892106 OK. Thanks. Will send out a follow-up pr. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14879] [SQL] Move CreateMetastoreDataSo...
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/12645#discussion_r60839199
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/createDataSourceTables.scala
---
@@ -0,0 +1,452 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command
+
+import java.util.regex.Pattern
+
+import scala.collection.mutable
+import scala.util.control.NonFatal
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql._
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.analysis.EliminateSubqueryAliases
+import org.apache.spark.sql.catalyst.catalog.{CatalogColumn,
CatalogStorageFormat, CatalogTable, CatalogTableType}
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.execution.datasources.{BucketSpec, DataSource,
HadoopFsRelation, LogicalRelation}
+import org.apache.spark.sql.internal.HiveSerDe
+import org.apache.spark.sql.sources.InsertableRelation
+import org.apache.spark.sql.types._
+
+/**
+ * A command used to create a data source table.
+ *
+ * Note: This is different from [[CreateTable]]. Please check the syntax
for difference.
+ * This is not intended for temporary tables.
+ *
+ * The syntax of using this command in SQL is:
+ * {{{
+ * CREATE TABLE [IF NOT EXISTS] [db_name.]table_name
+ * [(col1 data_type [COMMENT col_comment], ...)]
+ * USING format OPTIONS ([option1_name "option1_value", option2_name
"option2_value", ...])
+ * }}}
+ */
+case class CreateDataSourceTableCommand(
+tableIdent: TableIdentifier,
+userSpecifiedSchema: Option[StructType],
+provider: String,
+options: Map[String, String],
+ignoreIfExists: Boolean,
+managedIfNoPath: Boolean)
+ extends RunnableCommand {
+
+ override def run(sqlContext: SQLContext): Seq[Row] = {
+// Since we are saving metadata to metastore, we need to check if
metastore supports
+// the table name and database name we have for this query.
MetaStoreUtils.validateName
+// is the method used by Hive to check if a table name or a database
name is valid for
+// the metastore.
+if (!CreateDataSourceTableUtils.validateName(tableIdent.table)) {
+ throw new AnalysisException(s"Table name ${tableIdent.table} is not
a valid name for " +
+s"metastore. Metastore only accepts table name containing
characters, numbers and _.")
+}
+if (tableIdent.database.isDefined &&
+ !CreateDataSourceTableUtils.validateName(tableIdent.database.get)) {
+ throw new AnalysisException(s"Database name
${tableIdent.database.get} is not a valid name " +
+s"for metastore. Metastore only accepts database name containing "
+
+s"characters, numbers and _.")
+}
+
+val tableName = tableIdent.unquotedString
+val sessionState = sqlContext.sessionState
+
+if (sessionState.catalog.tableExists(tableIdent)) {
+ if (ignoreIfExists) {
+return Seq.empty[Row]
+ } else {
+throw new AnalysisException(s"Table $tableName already exists.")
+ }
+}
+
+var isExternal = true
+val optionsWithPath =
+ if (!options.contains("path") && managedIfNoPath) {
+isExternal = false
+options + ("path" ->
sessionState.catalog.defaultTablePath(tableIdent))
+ } else {
+options
+ }
+
+// Create the relation to validate the arguments before writing the
metadata to the metastore.
+DataSource(
+ sqlContext = sqlContext,
+ userSpecifiedSchema = userSpecifiedSchema,
+ className = provider,
+ bucketSpec = None,
+ options = optionsWithPath).resolveRelation()
+
+CreateDataSourceTableUtils.createDataSourceTable(
+ sqlContext = sqlContext,
[GitHub] spark pull request: [SPARK-14879] [SQL] Move CreateMetastoreDataSo...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/12645#issuecomment-213891570 Most of the problems I pointed out also existed in the old code, so feel free to merge this one and submit a follow-up pr to address them. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14879] [SQL] Move CreateMetastoreDataSo...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/12645#discussion_r60839135
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/createDataSourceTables.scala
---
@@ -0,0 +1,452 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command
+
+import java.util.regex.Pattern
+
+import scala.collection.mutable
+import scala.util.control.NonFatal
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql._
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.analysis.EliminateSubqueryAliases
+import org.apache.spark.sql.catalyst.catalog.{CatalogColumn,
CatalogStorageFormat, CatalogTable, CatalogTableType}
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.execution.datasources.{BucketSpec, DataSource,
HadoopFsRelation, LogicalRelation}
+import org.apache.spark.sql.internal.HiveSerDe
+import org.apache.spark.sql.sources.InsertableRelation
+import org.apache.spark.sql.types._
+
+/**
+ * A command used to create a data source table.
+ *
+ * Note: This is different from [[CreateTable]]. Please check the syntax
for difference.
+ * This is not intended for temporary tables.
+ *
+ * The syntax of using this command in SQL is:
+ * {{{
+ * CREATE TABLE [IF NOT EXISTS] [db_name.]table_name
+ * [(col1 data_type [COMMENT col_comment], ...)]
+ * USING format OPTIONS ([option1_name "option1_value", option2_name
"option2_value", ...])
+ * }}}
+ */
+case class CreateDataSourceTableCommand(
+tableIdent: TableIdentifier,
+userSpecifiedSchema: Option[StructType],
+provider: String,
+options: Map[String, String],
+ignoreIfExists: Boolean,
+managedIfNoPath: Boolean)
+ extends RunnableCommand {
+
+ override def run(sqlContext: SQLContext): Seq[Row] = {
+// Since we are saving metadata to metastore, we need to check if
metastore supports
+// the table name and database name we have for this query.
MetaStoreUtils.validateName
+// is the method used by Hive to check if a table name or a database
name is valid for
+// the metastore.
+if (!CreateDataSourceTableUtils.validateName(tableIdent.table)) {
+ throw new AnalysisException(s"Table name ${tableIdent.table} is not
a valid name for " +
+s"metastore. Metastore only accepts table name containing
characters, numbers and _.")
+}
+if (tableIdent.database.isDefined &&
+ !CreateDataSourceTableUtils.validateName(tableIdent.database.get)) {
+ throw new AnalysisException(s"Database name
${tableIdent.database.get} is not a valid name " +
+s"for metastore. Metastore only accepts database name containing "
+
+s"characters, numbers and _.")
+}
+
+val tableName = tableIdent.unquotedString
+val sessionState = sqlContext.sessionState
+
+if (sessionState.catalog.tableExists(tableIdent)) {
+ if (ignoreIfExists) {
+return Seq.empty[Row]
+ } else {
+throw new AnalysisException(s"Table $tableName already exists.")
+ }
+}
+
+var isExternal = true
+val optionsWithPath =
+ if (!options.contains("path") && managedIfNoPath) {
+isExternal = false
+options + ("path" ->
sessionState.catalog.defaultTablePath(tableIdent))
+ } else {
+options
+ }
+
+// Create the relation to validate the arguments before writing the
metadata to the metastore.
+DataSource(
+ sqlContext = sqlContext,
+ userSpecifiedSchema = userSpecifiedSchema,
+ className = provider,
+ bucketSpec = None,
+ options = optionsWithPath).resolveRelation()
+
+CreateDataSourceTableUtils.createDataSourceTable(
+ sqlContext = sqlContext,
[GitHub] spark pull request: [SPARK-14879] [SQL] Move CreateMetastoreDataSo...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/12645#discussion_r60839130
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/createDataSourceTables.scala
---
@@ -0,0 +1,452 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command
+
+import java.util.regex.Pattern
+
+import scala.collection.mutable
+import scala.util.control.NonFatal
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql._
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.analysis.EliminateSubqueryAliases
+import org.apache.spark.sql.catalyst.catalog.{CatalogColumn,
CatalogStorageFormat, CatalogTable, CatalogTableType}
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.execution.datasources.{BucketSpec, DataSource,
HadoopFsRelation, LogicalRelation}
+import org.apache.spark.sql.internal.HiveSerDe
+import org.apache.spark.sql.sources.InsertableRelation
+import org.apache.spark.sql.types._
+
+/**
+ * A command used to create a data source table.
+ *
+ * Note: This is different from [[CreateTable]]. Please check the syntax
for difference.
+ * This is not intended for temporary tables.
+ *
+ * The syntax of using this command in SQL is:
+ * {{{
+ * CREATE TABLE [IF NOT EXISTS] [db_name.]table_name
+ * [(col1 data_type [COMMENT col_comment], ...)]
+ * USING format OPTIONS ([option1_name "option1_value", option2_name
"option2_value", ...])
+ * }}}
+ */
+case class CreateDataSourceTableCommand(
+tableIdent: TableIdentifier,
+userSpecifiedSchema: Option[StructType],
+provider: String,
+options: Map[String, String],
+ignoreIfExists: Boolean,
+managedIfNoPath: Boolean)
+ extends RunnableCommand {
+
+ override def run(sqlContext: SQLContext): Seq[Row] = {
+// Since we are saving metadata to metastore, we need to check if
metastore supports
+// the table name and database name we have for this query.
MetaStoreUtils.validateName
+// is the method used by Hive to check if a table name or a database
name is valid for
+// the metastore.
+if (!CreateDataSourceTableUtils.validateName(tableIdent.table)) {
+ throw new AnalysisException(s"Table name ${tableIdent.table} is not
a valid name for " +
+s"metastore. Metastore only accepts table name containing
characters, numbers and _.")
+}
+if (tableIdent.database.isDefined &&
+ !CreateDataSourceTableUtils.validateName(tableIdent.database.get)) {
+ throw new AnalysisException(s"Database name
${tableIdent.database.get} is not a valid name " +
+s"for metastore. Metastore only accepts database name containing "
+
+s"characters, numbers and _.")
+}
+
+val tableName = tableIdent.unquotedString
+val sessionState = sqlContext.sessionState
+
+if (sessionState.catalog.tableExists(tableIdent)) {
+ if (ignoreIfExists) {
+return Seq.empty[Row]
+ } else {
+throw new AnalysisException(s"Table $tableName already exists.")
+ }
+}
+
+var isExternal = true
+val optionsWithPath =
+ if (!options.contains("path") && managedIfNoPath) {
+isExternal = false
+options + ("path" ->
sessionState.catalog.defaultTablePath(tableIdent))
+ } else {
+options
+ }
+
+// Create the relation to validate the arguments before writing the
metadata to the metastore.
+DataSource(
+ sqlContext = sqlContext,
+ userSpecifiedSchema = userSpecifiedSchema,
+ className = provider,
+ bucketSpec = None,
+ options = optionsWithPath).resolveRelation()
+
+CreateDataSourceTableUtils.createDataSourceTable(
+ sqlContext = sqlContext,
[GitHub] spark pull request: [SPARK-14879] [SQL] Move CreateMetastoreDataSo...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/12645#discussion_r60839110
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/createDataSourceTables.scala
---
@@ -0,0 +1,452 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command
+
+import java.util.regex.Pattern
+
+import scala.collection.mutable
+import scala.util.control.NonFatal
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql._
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.analysis.EliminateSubqueryAliases
+import org.apache.spark.sql.catalyst.catalog.{CatalogColumn,
CatalogStorageFormat, CatalogTable, CatalogTableType}
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.execution.datasources.{BucketSpec, DataSource,
HadoopFsRelation, LogicalRelation}
+import org.apache.spark.sql.internal.HiveSerDe
+import org.apache.spark.sql.sources.InsertableRelation
+import org.apache.spark.sql.types._
+
+/**
+ * A command used to create a data source table.
+ *
+ * Note: This is different from [[CreateTable]]. Please check the syntax
for difference.
+ * This is not intended for temporary tables.
+ *
+ * The syntax of using this command in SQL is:
+ * {{{
+ * CREATE TABLE [IF NOT EXISTS] [db_name.]table_name
+ * [(col1 data_type [COMMENT col_comment], ...)]
+ * USING format OPTIONS ([option1_name "option1_value", option2_name
"option2_value", ...])
+ * }}}
+ */
+case class CreateDataSourceTableCommand(
+tableIdent: TableIdentifier,
+userSpecifiedSchema: Option[StructType],
+provider: String,
+options: Map[String, String],
+ignoreIfExists: Boolean,
+managedIfNoPath: Boolean)
+ extends RunnableCommand {
+
+ override def run(sqlContext: SQLContext): Seq[Row] = {
+// Since we are saving metadata to metastore, we need to check if
metastore supports
+// the table name and database name we have for this query.
MetaStoreUtils.validateName
+// is the method used by Hive to check if a table name or a database
name is valid for
+// the metastore.
+if (!CreateDataSourceTableUtils.validateName(tableIdent.table)) {
+ throw new AnalysisException(s"Table name ${tableIdent.table} is not
a valid name for " +
+s"metastore. Metastore only accepts table name containing
characters, numbers and _.")
+}
+if (tableIdent.database.isDefined &&
+ !CreateDataSourceTableUtils.validateName(tableIdent.database.get)) {
+ throw new AnalysisException(s"Database name
${tableIdent.database.get} is not a valid name " +
+s"for metastore. Metastore only accepts database name containing "
+
+s"characters, numbers and _.")
+}
+
+val tableName = tableIdent.unquotedString
+val sessionState = sqlContext.sessionState
+
+if (sessionState.catalog.tableExists(tableIdent)) {
+ if (ignoreIfExists) {
+return Seq.empty[Row]
+ } else {
+throw new AnalysisException(s"Table $tableName already exists.")
+ }
+}
+
+var isExternal = true
+val optionsWithPath =
+ if (!options.contains("path") && managedIfNoPath) {
+isExternal = false
+options + ("path" ->
sessionState.catalog.defaultTablePath(tableIdent))
+ } else {
+options
+ }
+
+// Create the relation to validate the arguments before writing the
metadata to the metastore.
+DataSource(
+ sqlContext = sqlContext,
+ userSpecifiedSchema = userSpecifiedSchema,
+ className = provider,
+ bucketSpec = None,
+ options = optionsWithPath).resolveRelation()
+
+CreateDataSourceTableUtils.createDataSourceTable(
+ sqlContext = sqlContext,
[GitHub] spark pull request: [SPARK-14879] [SQL] Move CreateMetastoreDataSo...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/12645#discussion_r60839082
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/createDataSourceTables.scala
---
@@ -0,0 +1,452 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command
+
+import java.util.regex.Pattern
+
+import scala.collection.mutable
+import scala.util.control.NonFatal
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql._
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.analysis.EliminateSubqueryAliases
+import org.apache.spark.sql.catalyst.catalog.{CatalogColumn,
CatalogStorageFormat, CatalogTable, CatalogTableType}
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.execution.datasources.{BucketSpec, DataSource,
HadoopFsRelation, LogicalRelation}
+import org.apache.spark.sql.internal.HiveSerDe
+import org.apache.spark.sql.sources.InsertableRelation
+import org.apache.spark.sql.types._
+
+/**
+ * A command used to create a data source table.
+ *
+ * Note: This is different from [[CreateTable]]. Please check the syntax
for difference.
+ * This is not intended for temporary tables.
+ *
+ * The syntax of using this command in SQL is:
+ * {{{
+ * CREATE TABLE [IF NOT EXISTS] [db_name.]table_name
+ * [(col1 data_type [COMMENT col_comment], ...)]
+ * USING format OPTIONS ([option1_name "option1_value", option2_name
"option2_value", ...])
+ * }}}
+ */
+case class CreateDataSourceTableCommand(
+tableIdent: TableIdentifier,
+userSpecifiedSchema: Option[StructType],
+provider: String,
+options: Map[String, String],
+ignoreIfExists: Boolean,
+managedIfNoPath: Boolean)
+ extends RunnableCommand {
+
+ override def run(sqlContext: SQLContext): Seq[Row] = {
+// Since we are saving metadata to metastore, we need to check if
metastore supports
+// the table name and database name we have for this query.
MetaStoreUtils.validateName
+// is the method used by Hive to check if a table name or a database
name is valid for
+// the metastore.
+if (!CreateDataSourceTableUtils.validateName(tableIdent.table)) {
+ throw new AnalysisException(s"Table name ${tableIdent.table} is not
a valid name for " +
+s"metastore. Metastore only accepts table name containing
characters, numbers and _.")
+}
+if (tableIdent.database.isDefined &&
+ !CreateDataSourceTableUtils.validateName(tableIdent.database.get)) {
+ throw new AnalysisException(s"Database name
${tableIdent.database.get} is not a valid name " +
+s"for metastore. Metastore only accepts database name containing "
+
+s"characters, numbers and _.")
+}
+
+val tableName = tableIdent.unquotedString
+val sessionState = sqlContext.sessionState
+
+if (sessionState.catalog.tableExists(tableIdent)) {
+ if (ignoreIfExists) {
+return Seq.empty[Row]
+ } else {
+throw new AnalysisException(s"Table $tableName already exists.")
+ }
+}
+
+var isExternal = true
+val optionsWithPath =
+ if (!options.contains("path") && managedIfNoPath) {
+isExternal = false
+options + ("path" ->
sessionState.catalog.defaultTablePath(tableIdent))
+ } else {
+options
+ }
+
+// Create the relation to validate the arguments before writing the
metadata to the metastore.
+DataSource(
+ sqlContext = sqlContext,
+ userSpecifiedSchema = userSpecifiedSchema,
+ className = provider,
+ bucketSpec = None,
+ options = optionsWithPath).resolveRelation()
+
+CreateDataSourceTableUtils.createDataSourceTable(
+ sqlContext = sqlContext,
[GitHub] spark pull request: [SPARK-13902][SPARK-14269][SCHEDULER] Eliminat...
Github user kayousterhout commented on the pull request: https://github.com/apache/spark/pull/12060#issuecomment-213890432 In theory this seems like a good idea: there do seem to be lots of unnecessary calls to submitWaitingStages(), and those calls seem somewhat expensive since they repeatedly re-compute the stage DAG. I'm a little nervous about the change though, because the DAGScheduler code is generally somewhat brittle and doesn't have well-documented invariants / abstractions, so this should be merged towards the beginning of a release cycle, ideally. I'm most nervous about the change to getAncestorShuffleDependencies (which adds a lot of complexity) so curious to hear why that is necessary. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14879] [SQL] Move CreateMetastoreDataSo...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/12645#discussion_r60839067
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/createDataSourceTables.scala
---
@@ -0,0 +1,452 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command
+
+import java.util.regex.Pattern
+
+import scala.collection.mutable
+import scala.util.control.NonFatal
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql._
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.analysis.EliminateSubqueryAliases
+import org.apache.spark.sql.catalyst.catalog.{CatalogColumn,
CatalogStorageFormat, CatalogTable, CatalogTableType}
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.execution.datasources.{BucketSpec, DataSource,
HadoopFsRelation, LogicalRelation}
+import org.apache.spark.sql.internal.HiveSerDe
+import org.apache.spark.sql.sources.InsertableRelation
+import org.apache.spark.sql.types._
+
+/**
+ * A command used to create a data source table.
+ *
+ * Note: This is different from [[CreateTable]]. Please check the syntax
for difference.
+ * This is not intended for temporary tables.
+ *
+ * The syntax of using this command in SQL is:
+ * {{{
+ * CREATE TABLE [IF NOT EXISTS] [db_name.]table_name
+ * [(col1 data_type [COMMENT col_comment], ...)]
+ * USING format OPTIONS ([option1_name "option1_value", option2_name
"option2_value", ...])
+ * }}}
+ */
+case class CreateDataSourceTableCommand(
+tableIdent: TableIdentifier,
+userSpecifiedSchema: Option[StructType],
+provider: String,
+options: Map[String, String],
+ignoreIfExists: Boolean,
+managedIfNoPath: Boolean)
+ extends RunnableCommand {
+
+ override def run(sqlContext: SQLContext): Seq[Row] = {
+// Since we are saving metadata to metastore, we need to check if
metastore supports
+// the table name and database name we have for this query.
MetaStoreUtils.validateName
+// is the method used by Hive to check if a table name or a database
name is valid for
+// the metastore.
+if (!CreateDataSourceTableUtils.validateName(tableIdent.table)) {
+ throw new AnalysisException(s"Table name ${tableIdent.table} is not
a valid name for " +
+s"metastore. Metastore only accepts table name containing
characters, numbers and _.")
+}
+if (tableIdent.database.isDefined &&
+ !CreateDataSourceTableUtils.validateName(tableIdent.database.get)) {
+ throw new AnalysisException(s"Database name
${tableIdent.database.get} is not a valid name " +
+s"for metastore. Metastore only accepts database name containing "
+
+s"characters, numbers and _.")
+}
+
+val tableName = tableIdent.unquotedString
+val sessionState = sqlContext.sessionState
+
+if (sessionState.catalog.tableExists(tableIdent)) {
+ if (ignoreIfExists) {
+return Seq.empty[Row]
+ } else {
+throw new AnalysisException(s"Table $tableName already exists.")
+ }
+}
+
+var isExternal = true
+val optionsWithPath =
+ if (!options.contains("path") && managedIfNoPath) {
+isExternal = false
+options + ("path" ->
sessionState.catalog.defaultTablePath(tableIdent))
+ } else {
+options
+ }
+
+// Create the relation to validate the arguments before writing the
metadata to the metastore.
+DataSource(
+ sqlContext = sqlContext,
+ userSpecifiedSchema = userSpecifiedSchema,
+ className = provider,
+ bucketSpec = None,
+ options = optionsWithPath).resolveRelation()
+
+CreateDataSourceTableUtils.createDataSourceTable(
+ sqlContext = sqlContext,
[GitHub] spark pull request: [SPARK-14879] [SQL] Move CreateMetastoreDataSo...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/12645#discussion_r60838889
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/createDataSourceTables.scala
---
@@ -0,0 +1,452 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command
+
+import java.util.regex.Pattern
+
+import scala.collection.mutable
+import scala.util.control.NonFatal
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql._
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.analysis.EliminateSubqueryAliases
+import org.apache.spark.sql.catalyst.catalog.{CatalogColumn,
CatalogStorageFormat, CatalogTable, CatalogTableType}
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.execution.datasources.{BucketSpec, DataSource,
HadoopFsRelation, LogicalRelation}
+import org.apache.spark.sql.internal.HiveSerDe
+import org.apache.spark.sql.sources.InsertableRelation
+import org.apache.spark.sql.types._
+
+/**
+ * A command used to create a data source table.
+ *
+ * Note: This is different from [[CreateTable]]. Please check the syntax
for difference.
+ * This is not intended for temporary tables.
+ *
+ * The syntax of using this command in SQL is:
+ * {{{
+ * CREATE TABLE [IF NOT EXISTS] [db_name.]table_name
+ * [(col1 data_type [COMMENT col_comment], ...)]
+ * USING format OPTIONS ([option1_name "option1_value", option2_name
"option2_value", ...])
+ * }}}
+ */
+case class CreateDataSourceTableCommand(
+tableIdent: TableIdentifier,
+userSpecifiedSchema: Option[StructType],
+provider: String,
+options: Map[String, String],
+ignoreIfExists: Boolean,
+managedIfNoPath: Boolean)
+ extends RunnableCommand {
+
+ override def run(sqlContext: SQLContext): Seq[Row] = {
+// Since we are saving metadata to metastore, we need to check if
metastore supports
+// the table name and database name we have for this query.
MetaStoreUtils.validateName
+// is the method used by Hive to check if a table name or a database
name is valid for
+// the metastore.
+if (!CreateDataSourceTableUtils.validateName(tableIdent.table)) {
+ throw new AnalysisException(s"Table name ${tableIdent.table} is not
a valid name for " +
+s"metastore. Metastore only accepts table name containing
characters, numbers and _.")
+}
+if (tableIdent.database.isDefined &&
+ !CreateDataSourceTableUtils.validateName(tableIdent.database.get)) {
+ throw new AnalysisException(s"Database name
${tableIdent.database.get} is not a valid name " +
+s"for metastore. Metastore only accepts database name containing "
+
+s"characters, numbers and _.")
+}
+
+val tableName = tableIdent.unquotedString
+val sessionState = sqlContext.sessionState
+
+if (sessionState.catalog.tableExists(tableIdent)) {
+ if (ignoreIfExists) {
+return Seq.empty[Row]
+ } else {
+throw new AnalysisException(s"Table $tableName already exists.")
+ }
+}
+
+var isExternal = true
+val optionsWithPath =
+ if (!options.contains("path") && managedIfNoPath) {
+isExternal = false
+options + ("path" ->
sessionState.catalog.defaultTablePath(tableIdent))
+ } else {
+options
+ }
+
+// Create the relation to validate the arguments before writing the
metadata to the metastore.
+DataSource(
+ sqlContext = sqlContext,
+ userSpecifiedSchema = userSpecifiedSchema,
+ className = provider,
+ bucketSpec = None,
+ options = optionsWithPath).resolveRelation()
+
+CreateDataSourceTableUtils.createDataSourceTable(
+ sqlContext = sqlContext,
[GitHub] spark pull request: [SPARK-14879] [SQL] Move CreateMetastoreDataSo...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/12645#discussion_r60838884
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/createDataSourceTables.scala
---
@@ -0,0 +1,452 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command
+
+import java.util.regex.Pattern
+
+import scala.collection.mutable
+import scala.util.control.NonFatal
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql._
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.analysis.EliminateSubqueryAliases
+import org.apache.spark.sql.catalyst.catalog.{CatalogColumn,
CatalogStorageFormat, CatalogTable, CatalogTableType}
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.execution.datasources.{BucketSpec, DataSource,
HadoopFsRelation, LogicalRelation}
+import org.apache.spark.sql.internal.HiveSerDe
+import org.apache.spark.sql.sources.InsertableRelation
+import org.apache.spark.sql.types._
+
+/**
+ * A command used to create a data source table.
+ *
+ * Note: This is different from [[CreateTable]]. Please check the syntax
for difference.
+ * This is not intended for temporary tables.
+ *
+ * The syntax of using this command in SQL is:
+ * {{{
+ * CREATE TABLE [IF NOT EXISTS] [db_name.]table_name
+ * [(col1 data_type [COMMENT col_comment], ...)]
+ * USING format OPTIONS ([option1_name "option1_value", option2_name
"option2_value", ...])
+ * }}}
+ */
+case class CreateDataSourceTableCommand(
+tableIdent: TableIdentifier,
+userSpecifiedSchema: Option[StructType],
+provider: String,
+options: Map[String, String],
+ignoreIfExists: Boolean,
+managedIfNoPath: Boolean)
+ extends RunnableCommand {
+
+ override def run(sqlContext: SQLContext): Seq[Row] = {
+// Since we are saving metadata to metastore, we need to check if
metastore supports
+// the table name and database name we have for this query.
MetaStoreUtils.validateName
+// is the method used by Hive to check if a table name or a database
name is valid for
+// the metastore.
+if (!CreateDataSourceTableUtils.validateName(tableIdent.table)) {
+ throw new AnalysisException(s"Table name ${tableIdent.table} is not
a valid name for " +
+s"metastore. Metastore only accepts table name containing
characters, numbers and _.")
+}
+if (tableIdent.database.isDefined &&
+ !CreateDataSourceTableUtils.validateName(tableIdent.database.get)) {
+ throw new AnalysisException(s"Database name
${tableIdent.database.get} is not a valid name " +
+s"for metastore. Metastore only accepts database name containing "
+
+s"characters, numbers and _.")
+}
+
+val tableName = tableIdent.unquotedString
+val sessionState = sqlContext.sessionState
+
+if (sessionState.catalog.tableExists(tableIdent)) {
+ if (ignoreIfExists) {
+return Seq.empty[Row]
+ } else {
+throw new AnalysisException(s"Table $tableName already exists.")
+ }
+}
+
+var isExternal = true
+val optionsWithPath =
+ if (!options.contains("path") && managedIfNoPath) {
+isExternal = false
+options + ("path" ->
sessionState.catalog.defaultTablePath(tableIdent))
+ } else {
+options
+ }
+
+// Create the relation to validate the arguments before writing the
metadata to the metastore.
+DataSource(
+ sqlContext = sqlContext,
+ userSpecifiedSchema = userSpecifiedSchema,
+ className = provider,
+ bucketSpec = None,
+ options = optionsWithPath).resolveRelation()
+
+CreateDataSourceTableUtils.createDataSourceTable(
+ sqlContext = sqlContext,
[GitHub] spark pull request: [SPARK-14833][SQL][STREAMING][TEST] Refactor S...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/12592 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14879] [SQL] Move CreateMetastoreDataSo...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/12645#discussion_r60838834
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/createDataSourceTables.scala
---
@@ -0,0 +1,452 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command
+
+import java.util.regex.Pattern
+
+import scala.collection.mutable
+import scala.util.control.NonFatal
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql._
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.analysis.EliminateSubqueryAliases
+import org.apache.spark.sql.catalyst.catalog.{CatalogColumn,
CatalogStorageFormat, CatalogTable, CatalogTableType}
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.execution.datasources.{BucketSpec, DataSource,
HadoopFsRelation, LogicalRelation}
+import org.apache.spark.sql.internal.HiveSerDe
+import org.apache.spark.sql.sources.InsertableRelation
+import org.apache.spark.sql.types._
+
+/**
+ * A command used to create a data source table.
+ *
+ * Note: This is different from [[CreateTable]]. Please check the syntax
for difference.
+ * This is not intended for temporary tables.
+ *
+ * The syntax of using this command in SQL is:
+ * {{{
+ * CREATE TABLE [IF NOT EXISTS] [db_name.]table_name
+ * [(col1 data_type [COMMENT col_comment], ...)]
+ * USING format OPTIONS ([option1_name "option1_value", option2_name
"option2_value", ...])
+ * }}}
+ */
+case class CreateDataSourceTableCommand(
+tableIdent: TableIdentifier,
+userSpecifiedSchema: Option[StructType],
+provider: String,
+options: Map[String, String],
+ignoreIfExists: Boolean,
+managedIfNoPath: Boolean)
+ extends RunnableCommand {
+
+ override def run(sqlContext: SQLContext): Seq[Row] = {
+// Since we are saving metadata to metastore, we need to check if
metastore supports
+// the table name and database name we have for this query.
MetaStoreUtils.validateName
+// is the method used by Hive to check if a table name or a database
name is valid for
+// the metastore.
+if (!CreateDataSourceTableUtils.validateName(tableIdent.table)) {
+ throw new AnalysisException(s"Table name ${tableIdent.table} is not
a valid name for " +
+s"metastore. Metastore only accepts table name containing
characters, numbers and _.")
--- End diff --
might be ok to leave the internal comment as metastore for now, but for
error messages let's say catalog.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14833][SQL][STREAMING][TEST] Refactor S...
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/12592#issuecomment-213889929 LGTM. Thanks, merging to master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13902][SPARK-14269][SCHEDULER] Eliminat...
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/12060#discussion_r60838822
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -403,32 +403,47 @@ class DAGScheduler(
parents.toList
}
- /** Find ancestor shuffle dependencies that are not registered in
shuffleToMapStage yet */
- private def getAncestorShuffleDependencies(rdd: RDD[_]):
Stack[ShuffleDependency[_, _, _]] = {
-val parents = new Stack[ShuffleDependency[_, _, _]]
+ /**
+ * Find ancestor shuffle dependencies that are not registered in
shuffleToMapStage yet.
+ * This is done in topological order to create ancestor stages first to
ensure that the result
+ * stage graph is correctly built.
+ */
+ private def getAncestorShuffleDependencies(rdd: RDD[_]):
Seq[ShuffleDependency[_, _, _]] = {
+val parents = new ArrayBuffer[ShuffleDependency[_, _, _]]
val visited = new HashSet[RDD[_]]
// We are manually maintaining a stack here to prevent
StackOverflowError
// caused by recursively visiting
val waitingForVisit = new Stack[RDD[_]]
def visit(r: RDD[_]) {
- if (!visited(r)) {
-visited += r
-for (dep <- r.dependencies) {
- dep match {
-case shufDep: ShuffleDependency[_, _, _] =>
- if (!shuffleToMapStage.contains(shufDep.shuffleId)) {
-parents.push(shufDep)
- }
-case _ =>
+ if (visited(r)) {
--- End diff --
Is the code change here fixing a correctness issue? Is this related to the
performance issue described in the pull request?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14879] [SQL] Move CreateMetastoreDataSo...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/12645#discussion_r60838810
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/createDataSourceTables.scala
---
@@ -0,0 +1,452 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command
+
+import java.util.regex.Pattern
+
+import scala.collection.mutable
+import scala.util.control.NonFatal
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql._
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.analysis.EliminateSubqueryAliases
+import org.apache.spark.sql.catalyst.catalog.{CatalogColumn,
CatalogStorageFormat, CatalogTable, CatalogTableType}
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.execution.datasources.{BucketSpec, DataSource,
HadoopFsRelation, LogicalRelation}
+import org.apache.spark.sql.internal.HiveSerDe
+import org.apache.spark.sql.sources.InsertableRelation
+import org.apache.spark.sql.types._
+
+/**
+ * A command used to create a data source table.
+ *
+ * Note: This is different from [[CreateTable]]. Please check the syntax
for difference.
+ * This is not intended for temporary tables.
+ *
+ * The syntax of using this command in SQL is:
+ * {{{
+ * CREATE TABLE [IF NOT EXISTS] [db_name.]table_name
+ * [(col1 data_type [COMMENT col_comment], ...)]
+ * USING format OPTIONS ([option1_name "option1_value", option2_name
"option2_value", ...])
+ * }}}
+ */
+case class CreateDataSourceTableCommand(
+tableIdent: TableIdentifier,
+userSpecifiedSchema: Option[StructType],
+provider: String,
+options: Map[String, String],
+ignoreIfExists: Boolean,
+managedIfNoPath: Boolean)
--- End diff --
when would managedIfNoPath ever be false?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14879] [SQL] Move CreateMetastoreDataSo...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/12645#discussion_r60838743
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/createDataSourceTables.scala
---
@@ -0,0 +1,452 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command
+
+import java.util.regex.Pattern
+
+import scala.collection.mutable
+import scala.util.control.NonFatal
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql._
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.analysis.EliminateSubqueryAliases
+import org.apache.spark.sql.catalyst.catalog.{CatalogColumn,
CatalogStorageFormat, CatalogTable, CatalogTableType}
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.execution.datasources.{BucketSpec, DataSource,
HadoopFsRelation, LogicalRelation}
+import org.apache.spark.sql.internal.HiveSerDe
+import org.apache.spark.sql.sources.InsertableRelation
+import org.apache.spark.sql.types._
+
+/**
+ * A command used to create a data source table.
+ *
+ * Note: This is different from [[CreateTable]]. Please check the syntax
for difference.
+ * This is not intended for temporary tables.
+ *
+ * The syntax of using this command in SQL is:
+ * {{{
+ * CREATE TABLE [IF NOT EXISTS] [db_name.]table_name
+ * [(col1 data_type [COMMENT col_comment], ...)]
+ * USING format OPTIONS ([option1_name "option1_value", option2_name
"option2_value", ...])
+ * }}}
+ */
+case class CreateDataSourceTableCommand(
+tableIdent: TableIdentifier,
+userSpecifiedSchema: Option[StructType],
+provider: String,
+options: Map[String, String],
+ignoreIfExists: Boolean,
+managedIfNoPath: Boolean)
+ extends RunnableCommand {
+
+ override def run(sqlContext: SQLContext): Seq[Row] = {
+// Since we are saving metadata to metastore, we need to check if
metastore supports
+// the table name and database name we have for this query.
MetaStoreUtils.validateName
+// is the method used by Hive to check if a table name or a database
name is valid for
+// the metastore.
+if (!CreateDataSourceTableUtils.validateName(tableIdent.table)) {
+ throw new AnalysisException(s"Table name ${tableIdent.table} is not
a valid name for " +
+s"metastore. Metastore only accepts table name containing
characters, numbers and _.")
--- End diff --
catalog.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14879] [SQL] Move CreateMetastoreDataSo...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/12645#discussion_r60838739
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/createDataSourceTables.scala
---
@@ -0,0 +1,452 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command
+
+import java.util.regex.Pattern
+
+import scala.collection.mutable
+import scala.util.control.NonFatal
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql._
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.analysis.EliminateSubqueryAliases
+import org.apache.spark.sql.catalyst.catalog.{CatalogColumn,
CatalogStorageFormat, CatalogTable, CatalogTableType}
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.execution.datasources.{BucketSpec, DataSource,
HadoopFsRelation, LogicalRelation}
+import org.apache.spark.sql.internal.HiveSerDe
+import org.apache.spark.sql.sources.InsertableRelation
+import org.apache.spark.sql.types._
+
+/**
+ * A command used to create a data source table.
+ *
+ * Note: This is different from [[CreateTable]]. Please check the syntax
for difference.
+ * This is not intended for temporary tables.
+ *
+ * The syntax of using this command in SQL is:
+ * {{{
+ * CREATE TABLE [IF NOT EXISTS] [db_name.]table_name
+ * [(col1 data_type [COMMENT col_comment], ...)]
+ * USING format OPTIONS ([option1_name "option1_value", option2_name
"option2_value", ...])
+ * }}}
+ */
+case class CreateDataSourceTableCommand(
+tableIdent: TableIdentifier,
+userSpecifiedSchema: Option[StructType],
+provider: String,
+options: Map[String, String],
+ignoreIfExists: Boolean,
+managedIfNoPath: Boolean)
+ extends RunnableCommand {
+
+ override def run(sqlContext: SQLContext): Seq[Row] = {
+// Since we are saving metadata to metastore, we need to check if
metastore supports
--- End diff --
metastore -> catalog; metastore is very hive specific
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14876][SQL] SparkSession should be case...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12643#issuecomment-213888417 **[Test build #56833 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56833/consoleFull)** for PR 12643 at commit [`40aefc0`](https://github.com/apache/spark/commit/40aefc081dcd11f610e2199d11c2002236280153). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11735] [SQL] Add a check in the constru...
Github user yhuai closed the pull request at: https://github.com/apache/spark/pull/9702 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP] [SQL] Towards removing HiveContext
Github user yhuai closed the pull request at: https://github.com/apache/spark/pull/12410 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SQL] How many tests will fail if we do not ha...
Github user yhuai closed the pull request at: https://github.com/apache/spark/pull/12367 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SQL] Can we remove executionHive?
Github user yhuai closed the pull request at: https://github.com/apache/spark/pull/12372 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14505] [Core] Fix bug : creating two Sp...
Github user the-sea commented on the pull request: https://github.com/apache/spark/pull/12273#issuecomment-213882250 @srowen sorry for the delay, I have free time only on weekends -_-! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14838][SQL] Set default size for ObjecT...
Github user viirya commented on the pull request: https://github.com/apache/spark/pull/12599#issuecomment-213881860 Unrelated failure. I think it is ok. Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14838][SQL] Set default size for ObjecT...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/12599 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14838][SQL] Set default size for ObjecT...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12599#issuecomment-213881827 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/56832/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14838][SQL] Set default size for ObjecT...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12599#issuecomment-213881826 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14838][SQL] Set default size for ObjecT...
Github user davies commented on the pull request: https://github.com/apache/spark/pull/12599#issuecomment-213881630 LGTM, Merging this into master, thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14487][SQL] User Defined Type registrat...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12259#issuecomment-213881371 **[Test build #56831 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56831/consoleFull)** for PR 12259 at commit [`06bdbc5`](https://github.com/apache/spark/commit/06bdbc518e7e85ce8627028c628d93f210f07cd8). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14098][SQL] Generate Java code that get...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11956#issuecomment-213881135 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14098][SQL] Generate Java code that get...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11956#issuecomment-213881140 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/56827/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14098][SQL] Generate Java code that get...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11956#issuecomment-213881027 **[Test build #56827 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56827/consoleFull)** for PR 11956 at commit [`cd8855f`](https://github.com/apache/spark/commit/cd8855f963d180ff9aec07509b192633ebe8856b). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14487][SQL] User Defined Type registrat...
Github user viirya commented on the pull request: https://github.com/apache/spark/pull/12259#issuecomment-213881020 retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14487][SQL] User Defined Type registrat...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12259#issuecomment-213880954 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/56828/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14838][SQL] Set default size for ObjecT...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12599#issuecomment-213880968 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/56825/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14838][SQL] Set default size for ObjecT...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12599#issuecomment-213880967 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14487][SQL] User Defined Type registrat...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12259#issuecomment-213880953 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14838][SQL] Set default size for ObjecT...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12599#issuecomment-213880939 **[Test build #56825 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56825/consoleFull)** for PR 12599 at commit [`c26c3bd`](https://github.com/apache/spark/commit/c26c3bd05e1ceb5459ae8e9c6ac4a4ae8c36f2fb). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14867][BUILD] Remove `--force` option i...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12631#issuecomment-213880951 **[Test build #56830 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56830/consoleFull)** for PR 12631 at commit [`8504cce`](https://github.com/apache/spark/commit/8504cce65231c38e70278c4c625aa3c87cb4c0c3). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14487][SQL] User Defined Type registrat...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12259#issuecomment-213880936 **[Test build #56828 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56828/consoleFull)** for PR 12259 at commit [`06bdbc5`](https://github.com/apache/spark/commit/06bdbc518e7e85ce8627028c628d93f210f07cd8). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14867][BUILD] Remove `--force` option i...
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12631#issuecomment-213880918 Rebased. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14874][SQL][Streaming] Remove the obsol...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12638#issuecomment-213879726 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14874][SQL][Streaming] Remove the obsol...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12638#issuecomment-213879728 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/56824/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14874][SQL][Streaming] Remove the obsol...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12638#issuecomment-213879699 **[Test build #56824 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56824/consoleFull)** for PR 12638 at commit [`c79cba9`](https://github.com/apache/spark/commit/c79cba9059b7ac2d6398c81b57ceece50b6b7526). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14867][BUILD] Remove `--force` option i...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12631#issuecomment-213877402 **[Test build #56829 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56829/consoleFull)** for PR 12631 at commit [`cfe9b4e`](https://github.com/apache/spark/commit/cfe9b4eb101b9497a76938967b931ba99bbe6c9f). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14487][SQL] User Defined Type registrat...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12259#issuecomment-213873676 **[Test build #56828 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56828/consoleFull)** for PR 12259 at commit [`06bdbc5`](https://github.com/apache/spark/commit/06bdbc518e7e85ce8627028c628d93f210f07cd8). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14487][SQL] User Defined Type registrat...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12259#issuecomment-213872213 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14487][SQL] User Defined Type registrat...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12259#issuecomment-213872211 **[Test build #56826 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56826/consoleFull)** for PR 12259 at commit [`3917f6b`](https://github.com/apache/spark/commit/3917f6bac8cec83cac50b2faf663f9d659876231). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14487][SQL] User Defined Type registrat...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12259#issuecomment-213872215 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/56826/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14487][SQL] User Defined Type registrat...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12259#issuecomment-213872166 **[Test build #56826 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56826/consoleFull)** for PR 12259 at commit [`3917f6b`](https://github.com/apache/spark/commit/3917f6bac8cec83cac50b2faf663f9d659876231). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14098][SQL] Generate Java code that get...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11956#issuecomment-213872164 **[Test build #56827 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56827/consoleFull)** for PR 11956 at commit [`cd8855f`](https://github.com/apache/spark/commit/cd8855f963d180ff9aec07509b192633ebe8856b). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14838][SQL] Set default size for ObjecT...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/12599#discussion_r60838031
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala
---
@@ -630,6 +630,29 @@ class DatasetSuite extends QueryTest with
SharedSQLContext {
// Make sure the generated code for this plan can compile and execute.
checkDataset(wideDF.map(_.getLong(0)), 0L until 10 : _*)
}
+
+ test("Estimate size on ObjectProducer will cause failure") {
--- End diff --
fix. thanks. Please see if the new name is more proper.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14838][SQL] Set default size for ObjecT...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12599#issuecomment-213870976 **[Test build #56825 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56825/consoleFull)** for PR 12599 at commit [`c26c3bd`](https://github.com/apache/spark/commit/c26c3bd05e1ceb5459ae8e9c6ac4a4ae8c36f2fb). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14874][SQL][Streaming] Remove the obsol...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12638#issuecomment-213870648 **[Test build #56824 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56824/consoleFull)** for PR 12638 at commit [`c79cba9`](https://github.com/apache/spark/commit/c79cba9059b7ac2d6398c81b57ceece50b6b7526). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14874][SQL][Streaming] Remove the obsol...
Github user lw-lin commented on the pull request: https://github.com/apache/spark/pull/12638#issuecomment-213870571 @marmbrus @tdas would you mind taking a look? Thanks! :-) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14487][SQL] User Defined Type registrat...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12259#issuecomment-213870537 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14487][SQL] User Defined Type registrat...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12259#issuecomment-213870536 **[Test build #56823 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56823/consoleFull)** for PR 12259 at commit [`023d281`](https://github.com/apache/spark/commit/023d281277e6fc6100f2bcb9f62a9f9fb67ce77b). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14487][SQL] User Defined Type registrat...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12259#issuecomment-213870539 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/56823/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14874][SQL][Streaming] Remove the obsol...
GitHub user lw-lin reopened a pull request: https://github.com/apache/spark/pull/12638 [SPARK-14874][SQL][Streaming] Remove the obsolete Batch representation ## What changes were proposed in this pull request? The `Batch` class, which had been used to indicate progress in a stream, was abandoned by [[SPARK-13985][SQL] Deterministic batches with ids](https://github.com/apache/spark/commit/caea15214571d9b12dcf1553e5c1cc8b83a8ba5b) and then became useless. This patch: - removes the `Batch` class - renames `getBatch(...)` to `getData(...)` for `Source`: - before [SPARK-13985](https://github.com/apache/spark/commit/caea15214571d9b12dcf1553e5c1cc8b83a8ba5b), it was: get**_NextBatch_**(start: Option[Offset]): **_Option[Batch]_** - after [SPARK-13985](https://github.com/apache/spark/commit/caea15214571d9b12dcf1553e5c1cc8b83a8ba5b), it became: get**_Batch_**(start: Option[Offset], end: Offset): **_DataFrame_** - proposed in this patch: get**_Data_**(start: Option[Offset], end: Offset): DataFrame - renames `addBatch(...)` to `addData(...)` for `Sink`: - before [SPARK-13985](https://github.com/apache/spark/commit/caea15214571d9b12dcf1553e5c1cc8b83a8ba5b), it was: addBatch(**_batch: Batch_**) - after [SPARK-13985](https://github.com/apache/spark/commit/caea15214571d9b12dcf1553e5c1cc8b83a8ba5b), it became: addBatch(batchId: Long, **_data: DataFrame_**) - proposed in this patch: add**_Data_**(batchId: Long, data: DataFrame) The renaming of public methods should be OK since they have not been in any release yet. ## How was this patch tested? N/A You can merge this pull request into a Git repository by running: $ git pull https://github.com/lw-lin/spark remove-batch Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/12638.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #12638 commit c79cba9059b7ac2d6398c81b57ceece50b6b7526 Author: Liwei Lin Date: 2016-04-23T10:15:51Z remove the useless Batch class --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14487][SQL] User Defined Type registrat...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12259#issuecomment-213870512 **[Test build #56823 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56823/consoleFull)** for PR 12259 at commit [`023d281`](https://github.com/apache/spark/commit/023d281277e6fc6100f2bcb9f62a9f9fb67ce77b). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14487][SQL] User Defined Type registrat...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/12259#discussion_r60837881
--- Diff:
mllib/src/test/scala/org/apache/spark/ml/linalg/MatrixUDTSuite.scala ---
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.linalg.udt
+
+import scala.beans.{BeanInfo, BeanProperty}
+
+import org.apache.spark.{SparkException, SparkFunSuite}
+import org.apache.spark.ml.linalg._
+import org.apache.spark.mllib.util.MLlibTestSparkContext
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.types._
+
+@BeanInfo
+private[ml] case class MyMatrixPoint(
+@BeanProperty label: Double,
+@BeanProperty matrix: Matrix)
+
+class MatrixUDTSuite extends SparkFunSuite with MLlibTestSparkContext {
--- End diff --
I need implicits from sqlContext to use `toDF`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14881][PYTHON][SPARKR] pyspark and spar...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12648#issuecomment-213864239 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14881][PYTHON][SPARKR] pyspark and spar...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12648#issuecomment-213864240 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/56822/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14881][PYTHON][SPARKR] pyspark and spar...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12648#issuecomment-213864100 **[Test build #56822 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56822/consoleFull)** for PR 12648 at commit [`005892a`](https://github.com/apache/spark/commit/005892acd63b5621306f3ffd36f596ea4f204357). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12148][SPARKR] fix doc after renaming D...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/12647 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12148][SPARKR] fix doc after renaming D...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/12647#issuecomment-213862448 Thanks - merging in master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14838][SQL] Set default size for ObjecT...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/12599#discussion_r60837461
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala
---
@@ -630,6 +630,29 @@ class DatasetSuite extends QueryTest with
SharedSQLContext {
// Make sure the generated code for this plan can compile and execute.
checkDataset(wideDF.map(_.getLong(0)), 0L until 10 : _*)
}
+
+ test("Estimate size on ObjectProducer will cause failure") {
--- End diff --
the test case name is wrong?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14838][SQL] Set default size for ObjecT...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12599#issuecomment-213862390 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14838][SQL] Set default size for ObjecT...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12599#issuecomment-213862394 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/56820/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14838][SQL] Set default size for ObjecT...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12599#issuecomment-213862360 **[Test build #56820 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56820/consoleFull)** for PR 12599 at commit [`6b6c12d`](https://github.com/apache/spark/commit/6b6c12d9f1287ec18df29629d528306c8c18d165). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
