[GitHub] spark pull request: [SPARK-15463][SQL] support creating dataframe ...
Github user xwu0226 commented on a diff in the pull request:
https://github.com/apache/spark/pull/13300#discussion_r64694941
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVRelation.scala
---
@@ -142,6 +145,75 @@ object CSVRelation extends Logging {
if (nonEmptyLines.hasNext) nonEmptyLines.drop(1)
}
}
+
+ def baseRdd(
+ sparkSession: SparkSession,
+ options: CSVOptions,
+ inputPaths: Seq[String]): RDD[String] = {
+readText(sparkSession, options, inputPaths.mkString(","))
+ }
+
+ def tokenRdd(
+ options: CSVOptions,
+ header: Array[String],
+ rdd: RDD[String]): RDD[Array[String]] = {
+val firstLine = if (options.headerFlag) findFirstLine(options, rdd)
else null
+univocityTokenizer(rdd, header, firstLine, options)
+ }
+
+ /**
+ * Returns the first line of the first non-empty file in path
+ */
+ def findFirstLine(options: CSVOptions, rdd: RDD[String]): String = {
+if (options.isCommentSet) {
+ val comment = options.comment.toString
+ rdd.filter { line =>
+line.trim.nonEmpty && !line.startsWith(comment)
+ }.first()
+} else {
+ rdd.filter { line =>
+line.trim.nonEmpty
+ }.first()
+}
+ }
+
+ def readText(
+ sparkSession: SparkSession,
+ options: CSVOptions,
+ location: String): RDD[String] = {
+if (Charset.forName(options.charset) == StandardCharsets.UTF_8) {
+ sparkSession.sparkContext.textFile(location)
+} else {
+ val charset = options.charset
+ sparkSession.sparkContext
+.hadoopFile[LongWritable, Text, TextInputFormat](location)
+.mapPartitions(_.map(pair => new String(pair._2.getBytes, 0,
pair._2.getLength, charset)))
+}
+ }
+
+ def verifySchema(schema: StructType): Unit = {
+schema.foreach { field =>
+ field.dataType match {
+case _: ArrayType | _: MapType | _: StructType =>
+ throw new UnsupportedOperationException(
+s"CSV data source does not support
${field.dataType.simpleString} data type.")
+case _ =>
+ }
+}
+ }
+
+ def getHeader(rdd: RDD[String], csvOptions: CSVOptions): Array[String] =
{
--- End diff --
This is also used in a few places to get the header from csv records.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8603][SPARKR] Use shell() instead of sy...
Github user sun-rui commented on a diff in the pull request:
https://github.com/apache/spark/pull/13165#discussion_r64694857
--- Diff: R/pkg/inst/tests/testthat/test_includeJAR.R ---
@@ -21,10 +21,13 @@ runScript <- function() {
sparkTestJarPath <- "R/lib/SparkR/test_support/sparktestjar_2.10-1.0.jar"
jarPath <- paste("--jars", shQuote(file.path(sparkHome,
sparkTestJarPath)))
scriptPath <- file.path(sparkHome,
"R/lib/SparkR/tests/testthat/jarTest.R")
- submitPath <- file.path(sparkHome, "bin/spark-submit")
- res <- system2(command = submitPath,
- args = c(jarPath, scriptPath),
- stdout = TRUE)
+ if (.Platform$OS.type == "windows") {
--- End diff --
you can call determineSparkSubmitBin() here
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15463][SQL] support creating dataframe ...
Github user xwu0226 commented on a diff in the pull request:
https://github.com/apache/spark/pull/13300#discussion_r64694834
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVInferSchema.scala
---
@@ -42,16 +42,23 @@ private[csv] object CSVInferSchema {
tokenRdd: RDD[Array[String]],
header: Array[String],
options: CSVOptions): StructType = {
-val startType: Array[DataType] =
Array.fill[DataType](header.length)(NullType)
-val rootTypes: Array[DataType] =
- tokenRdd.aggregate(startType)(inferRowType(options), mergeRowTypes)
+val structFields = if (options.inferSchemaFlag) {
--- End diff --
This method is used in both `csv.DefaultSource` and
`DataFrameReader.csv(ds: Dataset[String])`. So I refactored it here to take
care both the default schema type and `inferSchemaFlag=true` cases.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10372] [CORE] basic test framework for ...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/8559#issuecomment-221784338 on a related note, @squito can you in the future leave a msg indicating the branch a pr was merged once you merge it? There have been cases that lead to race conditions in merging and also mistakes in the branches that we needed to go back and audit. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10372] [CORE] basic test framework for ...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/8559#issuecomment-221784058 This is pretty cool! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP] [SPARK-8426] Enhance Blacklist mechanism...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13234#issuecomment-221783307 **[Test build #59344 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59344/consoleFull)** for PR 13234 at commit [`8f2534b`](https://github.com/apache/spark/commit/8f2534b1d4d90f1ed42c695a77f5a2fa588d3428). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10372] [CORE] basic test framework for ...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/8559 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15532] [SQL] Add SQLConf.ALLOW_MULTIPLE...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13310#issuecomment-221780558 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15532] [SQL] Add SQLConf.ALLOW_MULTIPLE...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13310#issuecomment-221780560 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59333/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15532] [SQL] Add SQLConf.ALLOW_MULTIPLE...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13310#issuecomment-221780470 **[Test build #59333 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59333/consoleFull)** for PR 13310 at commit [`f40a898`](https://github.com/apache/spark/commit/f40a89873ba92eaf5821dce4728d2aab84e1289e). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SQL] Prevent illegal NULL propagation when fi...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13290#issuecomment-221777604 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SQL] Prevent illegal NULL propagation when fi...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13290#issuecomment-221777607 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59334/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SQL] Prevent illegal NULL propagation when fi...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13290#issuecomment-221777507 **[Test build #59334 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59334/consoleFull)** for PR 13290 at commit [`127024d`](https://github.com/apache/spark/commit/127024da7e1058cd39b71e85c6dcd08b5e3e2b53). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15543][SQL] Rename DefaultSources to ma...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13311#issuecomment-221777001 **[Test build #59343 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59343/consoleFull)** for PR 13311 at commit [`94d6e7b`](https://github.com/apache/spark/commit/94d6e7b218e0a969b41f32bd61878cf890c3ba99). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15533][SQL]Deprecate Dataset.explode
Github user WeichenXu123 closed the pull request at: https://github.com/apache/spark/pull/13313 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15327] [SQL] fix split expression in wh...
Github user ueshin commented on the pull request: https://github.com/apache/spark/pull/13235#issuecomment-221776141 It looks like #12351 is the same issue about whole stage codegen with `splitExpressions`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15515] [SQL] Error Handling in Running ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13283#issuecomment-221775767 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59332/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15515] [SQL] Error Handling in Running ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13283#issuecomment-221775766 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15515] [SQL] Error Handling in Running ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13283#issuecomment-221775683 **[Test build #59332 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59332/consoleFull)** for PR 13283 at commit [`76f4f80`](https://github.com/apache/spark/commit/76f4f80f962e0271a2073a4cb8de0d513013cf87). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15542][SparkR] Make error message clear...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13308#issuecomment-221775528 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15542][SparkR] Make error message clear...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13308#issuecomment-221775529 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59342/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15542][SparkR] Make error message clear...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13308#issuecomment-221775481 **[Test build #59342 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59342/consoleFull)** for PR 13308 at commit [`cbd5163`](https://github.com/apache/spark/commit/cbd5163d73fa56a58e18598ece64aaa60e06cc1d). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-221774354 **[Test build #59341 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59341/consoleFull)** for PR 9192 at commit [`f67095e`](https://github.com/apache/spark/commit/f67095ef72540140aa2348b5262ffdf91685846a). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-221774407 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-221774409 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59341/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15542][SparkR] Make error message clear...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13308#issuecomment-221774053 **[Test build #59342 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59342/consoleFull)** for PR 13308 at commit [`cbd5163`](https://github.com/apache/spark/commit/cbd5163d73fa56a58e18598ece64aaa60e06cc1d). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [YARN][Doc][Minor] Remove several obsolete env...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13296#issuecomment-221773158 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59329/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [YARN][Doc][Minor] Remove several obsolete env...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13296#issuecomment-221773157 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [YARN][Doc][Minor] Remove several obsolete env...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13296#issuecomment-221773071 **[Test build #59329 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59329/consoleFull)** for PR 13296 at commit [`367e3b8`](https://github.com/apache/spark/commit/367e3b8de0633c100bc1a9bf4742f6af80ecfa68). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15542][SparkR] Make error message clear...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13308#issuecomment-221773031 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15542][SparkR] Make error message clear...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13308#issuecomment-221773032 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59340/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15542][SparkR] Make error message clear...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13308#issuecomment-221772977 **[Test build #59340 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59340/consoleFull)** for PR 13308 at commit [`88319c0`](https://github.com/apache/spark/commit/88319c022b8eb55f59f8080d488e30726f475580). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8603][SPARKR] Use shell() instead of sy...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13165#issuecomment-221772828 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-221772896 Thanks for the update. LGTM. Will merge after Jenkins passes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8603][SPARKR] Use shell() instead of sy...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13165#issuecomment-221772829 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59339/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8603][SPARKR] Use shell() instead of sy...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13165#issuecomment-221772778 **[Test build #59339 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59339/consoleFull)** for PR 13165 at commit [`0482ebb`](https://github.com/apache/spark/commit/0482ebbc43ff1bef8e7a6a16376c6ec36840a366). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/13284 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/13284#issuecomment-221772558 Yeah thats a good idea @wangmiao1981 can you open a JIRA to not mask `startsWith` and `endsWith` by updating our generics ? LGTM - Merging this to master and branch-2.0. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-221772591 **[Test build #59341 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59341/consoleFull)** for PR 9192 at commit [`f67095e`](https://github.com/apache/spark/commit/f67095ef72540140aa2348b5262ffdf91685846a). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15542][SparkR] Make error message clear...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13308#issuecomment-221772521 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59338/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15542][SparkR] Make error message clear...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13308#issuecomment-221772519 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15542][SparkR] Make error message clear...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13308#issuecomment-221772475 **[Test build #59338 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59338/consoleFull)** for PR 13308 at commit [`07806de`](https://github.com/apache/spark/commit/07806de09f4be0dd9501fe81684c07a45ad68672). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...
Github user felixcheung commented on the pull request: https://github.com/apache/spark/pull/13284#issuecomment-221772080 looks fine - I think we should really try to make startsWith and endsWith work though, but that could be a follow up. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15542][SparkR] Make error message clear...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13308#issuecomment-221771654 **[Test build #59340 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59340/consoleFull)** for PR 13308 at commit [`88319c0`](https://github.com/apache/spark/commit/88319c022b8eb55f59f8080d488e30726f475580). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15542][SparkR] Make error message clear...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13308#issuecomment-221771180 **[Test build #59338 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59338/consoleFull)** for PR 13308 at commit [`07806de`](https://github.com/apache/spark/commit/07806de09f4be0dd9501fe81684c07a45ad68672). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8603][SPARKR] Incorrect file separator ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13165#issuecomment-221771183 **[Test build #59339 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59339/consoleFull)** for PR 13165 at commit [`0482ebb`](https://github.com/apache/spark/commit/0482ebbc43ff1bef8e7a6a16376c6ec36840a366). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15515] [SQL] Error Handling in Running ...
Github user gatorsmile commented on the pull request: https://github.com/apache/spark/pull/13283#issuecomment-221770911 **Update**: The latest code changes contains - For JDBC format, we added an extra checking in the rule `ResolveRelations` of `Analyzer`. Without the PR, Spark will return the error message like: `Option 'url' not specified`. Now, we are reporting `Unsupported data source type for direct query on files: jdbc` - Make data source format name case incensitive so that error handling behaves consistent with the normal cases. - Added the test cases for all the supported formats. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8603][SPARKR] Incorrect file separator ...
Github user HyukjinKwon commented on the pull request: https://github.com/apache/spark/pull/13165#issuecomment-221770814 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8603][SPARKR] Incorrect file separator ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13165#issuecomment-221770536 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8603][SPARKR] Incorrect file separator ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13165#issuecomment-221770524 **[Test build #59336 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59336/consoleFull)** for PR 13165 at commit [`0482ebb`](https://github.com/apache/spark/commit/0482ebbc43ff1bef8e7a6a16376c6ec36840a366). * This patch **fails MiMa tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8603][SPARKR] Incorrect file separator ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13165#issuecomment-221770538 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59336/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-221770384 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59337/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-221770383 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SQL] Prevent illegal NULL propagation when fi...
Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/13290#discussion_r64688437
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
---
@@ -1448,6 +1450,37 @@ class Analyzer(
}
/**
+ * Fixes nullability of Attributes in a resolved LogicalPlan by using
the nullability of
+ * corresponding Attributes of its children output Attributes. This step
is needed because
+ * users can use a resolved AttributeReference in the Dataset API and
outer joins
+ * can change the nullability of an AttribtueReference. Without the fix,
a nullable column's
+ * nullable field can be actually set as non-nullable, which cause
illegal optimization
+ * (e.g., NULL propagation) and wrong answers.
+ * See SPARK-13484 and SPARK-13801 for the concrete queries of this case.
+ */
+ object FixNullability extends Rule[LogicalPlan] {
+
+def apply(plan: LogicalPlan): LogicalPlan = plan transformUp {
+ case q: LogicalPlan if q.resolved =>
+val childrenOutput = q.children.flatMap(c =>
c.output).groupBy(_.exprId).flatMap {
--- End diff --
yes, I got your point.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-221770380 **[Test build #59337 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59337/consoleFull)** for PR 9192 at commit [`90641a7`](https://github.com/apache/spark/commit/90641a71ff1860ddfe1a8e0bcb64cc0f0d2a56c6). * This patch **fails R style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-221770106 **[Test build #59337 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59337/consoleFull)** for PR 9192 at commit [`90641a7`](https://github.com/apache/spark/commit/90641a71ff1860ddfe1a8e0bcb64cc0f0d2a56c6). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15534][SPARK-15535][SQL] Truncate table...
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/13302#discussion_r64688076
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala ---
@@ -288,9 +288,10 @@ case class TruncateTableCommand(
override def run(sparkSession: SparkSession): Seq[Row] = {
val catalog = sparkSession.sessionState.catalog
if (!catalog.tableExists(tableName)) {
- logError(s"table '$tableName' in TRUNCATE TABLE does not exist.")
+ throw new AnalysisException(s"Table '$tableName' in TRUNCATE TABLE
does not exist.")
--- End diff --
Sorry Just realized that I had a typo `For drop table with IF
EXISTS keyword` should be `For drop table without IF EXISTS keyword`. Seems
without IF EXISTS keyword, Hive (I tried 1.2.1) does not throw an exception if
the table does not exist.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8603][SPARKR] Incorrect file separator ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13165#issuecomment-221769561 **[Test build #59336 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59336/consoleFull)** for PR 13165 at commit [`0482ebb`](https://github.com/apache/spark/commit/0482ebbc43ff1bef8e7a6a16376c6ec36840a366). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14204] [SQL] register driverClass rathe...
Github user zzcclp commented on the pull request: https://github.com/apache/spark/pull/12000#issuecomment-221769197 @JoshRosen , could you merge this into branch-1.6? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8603][SPARKR] Incorrect file separator ...
Github user HyukjinKwon commented on the pull request: https://github.com/apache/spark/pull/13165#issuecomment-221769185 I just addressed the comments and here is the (stdout)[https://gist.github.com/HyukjinKwon/4bf35184f3a30f3bce987a58ec2bbbab] of testing on Windiws. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15534][SPARK-15535][SQL] Truncate table...
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/13302#discussion_r64687912
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala ---
@@ -288,9 +288,10 @@ case class TruncateTableCommand(
override def run(sparkSession: SparkSession): Seq[Row] = {
val catalog = sparkSession.sessionState.catalog
if (!catalog.tableExists(tableName)) {
- logError(s"table '$tableName' in TRUNCATE TABLE does not exist.")
+ throw new AnalysisException(s"Table '$tableName' in TRUNCATE TABLE
does not exist.")
--- End diff --
@yhuai : Hive is doing the right thing of not throwing an exception when
`IF EXISTS` keyword is used in `DROP TABLE` (irrespective of the table being
there or not). If end user expects exception to be thrown, then one should not
use the `IF EXISTS` in the command. Having `IF EXISTS` helps for writing
pipelines which run daily and one wants "drop if exists or do nothing"
semantics.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-221768776 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-221768777 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59335/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-221768774 **[Test build #59335 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59335/consoleFull)** for PR 9192 at commit [`640ffca`](https://github.com/apache/spark/commit/640ffcaa2836d40337cda6c3ba1e51f30d16a44c). * This patch **fails R style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SQL] Prevent illegal NULL propagation when fi...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13290#issuecomment-221768479 **[Test build #59334 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59334/consoleFull)** for PR 13290 at commit [`127024d`](https://github.com/apache/spark/commit/127024da7e1058cd39b71e85c6dcd08b5e3e2b53). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-221768460 **[Test build #59335 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59335/consoleFull)** for PR 9192 at commit [`640ffca`](https://github.com/apache/spark/commit/640ffcaa2836d40337cda6c3ba1e51f30d16a44c). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15532] [SQL] Add SQLConf.ALLOW_MULTIPLE...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13310#issuecomment-221768478 **[Test build #59333 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59333/consoleFull)** for PR 13310 at commit [`f40a898`](https://github.com/apache/spark/commit/f40a89873ba92eaf5821dce4728d2aab84e1289e). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SQL] Prevent illegal NULL propagation when fi...
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/13290#discussion_r64687394
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
---
@@ -1448,6 +1450,37 @@ class Analyzer(
}
/**
+ * Fixes nullability of Attributes in a resolved LogicalPlan by using
the nullability of
+ * corresponding Attributes of its children output Attributes. This step
is needed because
+ * users can use a resolved AttributeReference in the Dataset API and
outer joins
+ * can change the nullability of an AttribtueReference. Without the fix,
a nullable column's
+ * nullable field can be actually set as non-nullable, which cause
illegal optimization
+ * (e.g., NULL propagation) and wrong answers.
+ * See SPARK-13484 and SPARK-13801 for the concrete queries of this case.
+ */
+ object FixNullability extends Rule[LogicalPlan] {
+
+def apply(plan: LogicalPlan): LogicalPlan = plan transformUp {
+ case q: LogicalPlan if q.resolved =>
+val childrenOutput = q.children.flatMap(c =>
c.output).groupBy(_.exprId).flatMap {
+ case (exprId, attributes) =>
+// If there are multiple Attributes having the same ExpirId,
we need to resolve
+// the conflict of nullable field.
+val nullable = attributes.map(_.nullable).reduce(_ || _)
--- End diff --
I feel it is not very possible. Let me think about it more.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15538][SPARK-15539][SQL] Truncate table...
Github user sureshthalamati commented on the pull request:
https://github.com/apache/spark/pull/13315#issuecomment-221768017
@andrewor14
I tried this PR. It does not seem to work/raise error if user attempts
to truncate on partition table with out specifying the partition spec.
val df = Seq((1 , "john", "CA") ,(2,"Mike", "NY"), (3, "Robert",
"CA")).toDF("id", "name", "state")
df.write.format("parquet").partitionBy("state").saveAsTable("emp16")
sql("truncate table emp16")
sql("select count(*) from emp16").show
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/9192#discussion_r64687334
--- Diff: R/pkg/inst/tests/testthat/test_sparkSQL.R ---
@@ -169,48 +169,50 @@ test_that("create DataFrame from RDD", {
error = function(err) {
skip("Hive is not build with SparkSQL, skipped")
})
- sql(hiveCtx, "CREATE TABLE people (name string, age double, height
float)")
- df <- read.df(hiveCtx, jsonPathNa, "json", schema)
+ assign(".sparkRHivesc", hiveCtx, envir = .sparkREnv)
--- End diff --
I'll refactor this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15515] [SQL] Error Handling in Running ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13283#issuecomment-221767896 **[Test build #59332 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59332/consoleFull)** for PR 13283 at commit [`76f4f80`](https://github.com/apache/spark/commit/76f4f80f962e0271a2073a4cb8de0d513013cf87). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user felixcheung commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-221767863 @shivaram it's true there's some scaffolding we need to add (though I'm pretty we could codegen them on the fly instead). I think the idea is this is temporary and in the next release (2.1.0?) we could remove these very easily (just a few lines before and after the method, plus renaming x.default back to x) to stop supporting deprecated calls. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-221767615 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-221767617 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59331/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-221767613 **[Test build #59331 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59331/consoleFull)** for PR 9192 at commit [`98e7ab9`](https://github.com/apache/spark/commit/98e7ab978dd1a7aba06ee47eb012c496f00cbe3b). * This patch **fails R style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-221767353 **[Test build #59331 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59331/consoleFull)** for PR 9192 at commit [`98e7ab9`](https://github.com/apache/spark/commit/98e7ab978dd1a7aba06ee47eb012c496f00cbe3b). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/9192#discussion_r64686844
--- Diff: R/pkg/R/SQLContext.R ---
@@ -362,12 +433,14 @@ sql <- function(sqlContext, sqlQuery) {
#' sc <- sparkR.init()
#' sqlContext <- sparkRSQL.init(sc)
#' path <- "path/to/file.json"
-#' df <- read.json(sqlContext, path)
+#' df <- read.json(path)
#' registerTempTable(df, "table")
-#' new_df <- tableToDF(sqlContext, "table")
+#' new_df <- tableToDF("table")
#' }
+#' @note since 2.0.0
--- End diff --
not yet, I don;t want to keep checking it, that would be the next PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15532] [SQL] Add SQLConf.ALLOW_MULTIPLE...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13310#issuecomment-221766492 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15532] [SQL] Add SQLConf.ALLOW_MULTIPLE...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13310#issuecomment-221766493 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59330/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15532] [SQL] Add SQLConf.ALLOW_MULTIPLE...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13310#issuecomment-221766483 **[Test build #59330 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59330/consoleFull)** for PR 13310 at commit [`bed21b0`](https://github.com/apache/spark/commit/bed21b01beecae4e8737ffef0630f5f9889c735f). * This patch **fails MiMa tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user sun-rui commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-221766378 LGTM except some minor comments. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/9192#discussion_r64686315
--- Diff: R/pkg/R/SQLContext.R ---
@@ -254,6 +301,7 @@ jsonFile <- function(sqlContext, path) {
#' df <- jsonRDD(sqlContext, rdd)
#'}
+# TODO: remove - this method is no longer exported
--- End diff --
opened SPARK-15545
one reason to not remove this right away is because we have been talking
about calling internal methods via `SparkR:::jsonRDD` for this and other RDD
methods.
we have .Deprecated() here though for a warning
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SQL] Prevent illegal NULL propagation when fi...
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/13290#discussion_r64686215
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
---
@@ -1448,6 +1450,37 @@ class Analyzer(
}
/**
+ * Fixes nullability of Attributes in a resolved LogicalPlan by using
the nullability of
+ * corresponding Attributes of its children output Attributes. This step
is needed because
+ * users can use a resolved AttributeReference in the Dataset API and
outer joins
+ * can change the nullability of an AttribtueReference. Without the fix,
a nullable column's
+ * nullable field can be actually set as non-nullable, which cause
illegal optimization
+ * (e.g., NULL propagation) and wrong answers.
+ * See SPARK-13484 and SPARK-13801 for the concrete queries of this case.
+ */
+ object FixNullability extends Rule[LogicalPlan] {
+
+def apply(plan: LogicalPlan): LogicalPlan = plan transformUp {
+ case q: LogicalPlan if q.resolved =>
+val childrenOutput = q.children.flatMap(c =>
c.output).groupBy(_.exprId).flatMap {
--- End diff --
Yea. This version tries to fix the nullability for the entire query plan
tree because I think it is possible to hit this problem in other cases when
using Dataset API.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user sun-rui commented on a diff in the pull request:
https://github.com/apache/spark/pull/9192#discussion_r64686229
--- Diff: R/pkg/R/jobj.R ---
@@ -77,6 +77,11 @@ print.jobj <- function(x, ...) {
cat("Java ref type", name, "id", x$id, "\n", sep = " ")
}
+getClassName.jobj <- function(x) {
--- End diff --
Could you address this comment?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15532] [SQL] Add SQLConf.ALLOW_MULTIPLE...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13310#issuecomment-221765755 **[Test build #59330 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59330/consoleFull)** for PR 13310 at commit [`bed21b0`](https://github.com/apache/spark/commit/bed21b01beecae4e8737ffef0630f5f9889c735f). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/9192#discussion_r64686152
--- Diff: R/pkg/R/SQLContext.R ---
@@ -37,6 +37,37 @@ getInternalType <- function(x) {
stop(paste("Unsupported type for SparkDataFrame:", class(x
}
+#' Temporary function to reroute old S3 Method call to new
+#' We need to check the class of x to ensure it is SQLContext before
dispatching
+dispatchFunc <- function(newFuncSig, x, ...) {
--- End diff --
I think this is very specific to this file - this helper is implemented to
specifically check & remove sqlContext parameter; I'll add more documentation
on this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15543][SQL] Rename DefaultSources to ma...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13311#issuecomment-221765444 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59326/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15543][SQL] Rename DefaultSources to ma...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13311#issuecomment-221765440 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15543][SQL] Rename DefaultSources to ma...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13311#issuecomment-221765335 **[Test build #59327 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59327/consoleFull)** for PR 13311 at commit [`51dcde9`](https://github.com/apache/spark/commit/51dcde9b1274b62fbd0a29c8cdfcb68d70a65f76). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15543][SQL] Rename DefaultSources to ma...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13311#issuecomment-221765416 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59327/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15543][SQL] Rename DefaultSources to ma...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13311#issuecomment-221765415 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15543][SQL] Rename DefaultSources to ma...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13311#issuecomment-221765364 **[Test build #59326 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59326/consoleFull)** for PR 13311 at commit [`eed27f0`](https://github.com/apache/spark/commit/eed27f035b1ffee9a64a1276e3fd34a0e8fe27f8). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [Spark-15129][R][DOC][WIP]R API changes in ML
Github user GayathriMurali commented on the pull request: https://github.com/apache/spark/pull/13285#issuecomment-221764817 @yanboliang Thanks, thats a good idea. However, that would just include example code and not how the output of summary() looks like. It might be useful to include that --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14634][ML] Add BisectingKMeansSummary
Github user zhengruifeng commented on the pull request: https://github.com/apache/spark/pull/12394#issuecomment-221764870 cc @mengxr --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user sun-rui commented on a diff in the pull request:
https://github.com/apache/spark/pull/9192#discussion_r64685560
--- Diff: R/pkg/R/SQLContext.R ---
@@ -362,12 +433,14 @@ sql <- function(sqlContext, sqlQuery) {
#' sc <- sparkR.init()
#' sqlContext <- sparkRSQL.init(sc)
#' path <- "path/to/file.json"
-#' df <- read.json(sqlContext, path)
+#' df <- read.json(path)
#' registerTempTable(df, "table")
-#' new_df <- tableToDF(sqlContext, "table")
+#' new_df <- tableToDF("table")
#' }
+#' @note since 2.0.0
--- End diff --
remove?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user sun-rui commented on a diff in the pull request:
https://github.com/apache/spark/pull/9192#discussion_r64685428
--- Diff: R/pkg/R/SQLContext.R ---
@@ -362,12 +433,14 @@ sql <- function(sqlContext, sqlQuery) {
#' sc <- sparkR.init()
#' sqlContext <- sparkRSQL.init(sc)
#' path <- "path/to/file.json"
-#' df <- read.json(sqlContext, path)
+#' df <- read.json(path)
#' registerTempTable(df, "table")
-#' new_df <- tableToDF(sqlContext, "table")
+#' new_df <- tableToDF("table")
#' }
+#' @note since 2.0.0
--- End diff --
?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15538][SQL] Adding error check for trun...
Github user sureshthalamati commented on the pull request: https://github.com/apache/spark/pull/13305#issuecomment-221762895 Thank you Andrew , Herman for your input. I am closing this pull request. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15538][SQL] Adding error check for trun...
Github user sureshthalamati closed the pull request at: https://github.com/apache/spark/pull/13305 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [YARN][Doc][Minor] Remove several obsolete env...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13296#issuecomment-221761576 **[Test build #59329 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59329/consoleFull)** for PR 13296 at commit [`367e3b8`](https://github.com/apache/spark/commit/367e3b8de0633c100bc1a9bf4742f6af80ecfa68). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15100][DOC] Modified user guide and exa...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13176#issuecomment-221761348 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
