[GitHub] spark pull request: [SPARK-15463][SQL] support creating dataframe ...

2016-05-25 Thread xwu0226
Github user xwu0226 commented on a diff in the pull request:

https://github.com/apache/spark/pull/13300#discussion_r64694941
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVRelation.scala
 ---
@@ -142,6 +145,75 @@ object CSVRelation extends Logging {
   if (nonEmptyLines.hasNext) nonEmptyLines.drop(1)
 }
   }
+
+  def baseRdd(
+  sparkSession: SparkSession,
+  options: CSVOptions,
+  inputPaths: Seq[String]): RDD[String] = {
+readText(sparkSession, options, inputPaths.mkString(","))
+  }
+
+  def tokenRdd(
+  options: CSVOptions,
+  header: Array[String],
+  rdd: RDD[String]): RDD[Array[String]] = {
+val firstLine = if (options.headerFlag) findFirstLine(options, rdd) 
else null
+univocityTokenizer(rdd, header, firstLine, options)
+  }
+
+  /**
+   * Returns the first line of the first non-empty file in path
+   */
+  def findFirstLine(options: CSVOptions, rdd: RDD[String]): String = {
+if (options.isCommentSet) {
+  val comment = options.comment.toString
+  rdd.filter { line =>
+line.trim.nonEmpty && !line.startsWith(comment)
+  }.first()
+} else {
+  rdd.filter { line =>
+line.trim.nonEmpty
+  }.first()
+}
+  }
+
+  def readText(
+  sparkSession: SparkSession,
+  options: CSVOptions,
+  location: String): RDD[String] = {
+if (Charset.forName(options.charset) == StandardCharsets.UTF_8) {
+  sparkSession.sparkContext.textFile(location)
+} else {
+  val charset = options.charset
+  sparkSession.sparkContext
+.hadoopFile[LongWritable, Text, TextInputFormat](location)
+.mapPartitions(_.map(pair => new String(pair._2.getBytes, 0, 
pair._2.getLength, charset)))
+}
+  }
+
+  def verifySchema(schema: StructType): Unit = {
+schema.foreach { field =>
+  field.dataType match {
+case _: ArrayType | _: MapType | _: StructType =>
+  throw new UnsupportedOperationException(
+s"CSV data source does not support 
${field.dataType.simpleString} data type.")
+case _ =>
+  }
+}
+  }
+
+  def getHeader(rdd: RDD[String], csvOptions: CSVOptions): Array[String] = 
{
--- End diff --

This is also used in a few places to get the header from csv records. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-8603][SPARKR] Use shell() instead of sy...

2016-05-25 Thread sun-rui
Github user sun-rui commented on a diff in the pull request:

https://github.com/apache/spark/pull/13165#discussion_r64694857
  
--- Diff: R/pkg/inst/tests/testthat/test_includeJAR.R ---
@@ -21,10 +21,13 @@ runScript <- function() {
   sparkTestJarPath <- "R/lib/SparkR/test_support/sparktestjar_2.10-1.0.jar"
   jarPath <- paste("--jars", shQuote(file.path(sparkHome, 
sparkTestJarPath)))
   scriptPath <- file.path(sparkHome, 
"R/lib/SparkR/tests/testthat/jarTest.R")
-  submitPath <- file.path(sparkHome, "bin/spark-submit")
-  res <- system2(command = submitPath,
- args = c(jarPath, scriptPath),
- stdout = TRUE)
+  if (.Platform$OS.type == "windows") {
--- End diff --

you can call determineSparkSubmitBin() here


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-15463][SQL] support creating dataframe ...

2016-05-25 Thread xwu0226
Github user xwu0226 commented on a diff in the pull request:

https://github.com/apache/spark/pull/13300#discussion_r64694834
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVInferSchema.scala
 ---
@@ -42,16 +42,23 @@ private[csv] object CSVInferSchema {
   tokenRdd: RDD[Array[String]],
   header: Array[String],
   options: CSVOptions): StructType = {
-val startType: Array[DataType] = 
Array.fill[DataType](header.length)(NullType)
-val rootTypes: Array[DataType] =
-  tokenRdd.aggregate(startType)(inferRowType(options), mergeRowTypes)
+val structFields = if (options.inferSchemaFlag) {
--- End diff --

This method is used in both `csv.DefaultSource` and 
`DataFrameReader.csv(ds: Dataset[String])`. So I refactored it here to take 
care both the default schema type and `inferSchemaFlag=true` cases.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-10372] [CORE] basic test framework for ...

2016-05-25 Thread rxin
Github user rxin commented on the pull request:

https://github.com/apache/spark/pull/8559#issuecomment-221784338
  
on a related note, @squito can you in the future leave a msg indicating the 
branch a pr was merged once you merge it? There have been cases that lead to 
race conditions in merging and also mistakes in the branches that we needed to 
go back and audit.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-10372] [CORE] basic test framework for ...

2016-05-25 Thread rxin
Github user rxin commented on the pull request:

https://github.com/apache/spark/pull/8559#issuecomment-221784058
  
This is pretty cool!



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [WIP] [SPARK-8426] Enhance Blacklist mechanism...

2016-05-25 Thread SparkQA
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13234#issuecomment-221783307
  
**[Test build #59344 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59344/consoleFull)**
 for PR 13234 at commit 
[`8f2534b`](https://github.com/apache/spark/commit/8f2534b1d4d90f1ed42c695a77f5a2fa588d3428).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-10372] [CORE] basic test framework for ...

2016-05-25 Thread asfgit
Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/8559


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-15532] [SQL] Add SQLConf.ALLOW_MULTIPLE...

2016-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13310#issuecomment-221780558
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-15532] [SQL] Add SQLConf.ALLOW_MULTIPLE...

2016-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13310#issuecomment-221780560
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59333/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-15532] [SQL] Add SQLConf.ALLOW_MULTIPLE...

2016-05-25 Thread SparkQA
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13310#issuecomment-221780470
  
**[Test build #59333 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59333/consoleFull)**
 for PR 13310 at commit 
[`f40a898`](https://github.com/apache/spark/commit/f40a89873ba92eaf5821dce4728d2aab84e1289e).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SQL] Prevent illegal NULL propagation when fi...

2016-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13290#issuecomment-221777604
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SQL] Prevent illegal NULL propagation when fi...

2016-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13290#issuecomment-221777607
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59334/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SQL] Prevent illegal NULL propagation when fi...

2016-05-25 Thread SparkQA
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13290#issuecomment-221777507
  
**[Test build #59334 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59334/consoleFull)**
 for PR 13290 at commit 
[`127024d`](https://github.com/apache/spark/commit/127024da7e1058cd39b71e85c6dcd08b5e3e2b53).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-15543][SQL] Rename DefaultSources to ma...

2016-05-25 Thread SparkQA
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13311#issuecomment-221777001
  
**[Test build #59343 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59343/consoleFull)**
 for PR 13311 at commit 
[`94d6e7b`](https://github.com/apache/spark/commit/94d6e7b218e0a969b41f32bd61878cf890c3ba99).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-15533][SQL]Deprecate Dataset.explode

2016-05-25 Thread WeichenXu123
Github user WeichenXu123 closed the pull request at:

https://github.com/apache/spark/pull/13313


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-15327] [SQL] fix split expression in wh...

2016-05-25 Thread ueshin
Github user ueshin commented on the pull request:

https://github.com/apache/spark/pull/13235#issuecomment-221776141
  
It looks like #12351 is the same issue about whole stage codegen with 
`splitExpressions`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-15515] [SQL] Error Handling in Running ...

2016-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13283#issuecomment-221775767
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59332/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-15515] [SQL] Error Handling in Running ...

2016-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13283#issuecomment-221775766
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-15515] [SQL] Error Handling in Running ...

2016-05-25 Thread SparkQA
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13283#issuecomment-221775683
  
**[Test build #59332 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59332/consoleFull)**
 for PR 13283 at commit 
[`76f4f80`](https://github.com/apache/spark/commit/76f4f80f962e0271a2073a4cb8de0d513013cf87).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-15542][SparkR] Make error message clear...

2016-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13308#issuecomment-221775528
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-15542][SparkR] Make error message clear...

2016-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13308#issuecomment-221775529
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59342/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-15542][SparkR] Make error message clear...

2016-05-25 Thread SparkQA
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13308#issuecomment-221775481
  
**[Test build #59342 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59342/consoleFull)**
 for PR 13308 at commit 
[`cbd5163`](https://github.com/apache/spark/commit/cbd5163d73fa56a58e18598ece64aaa60e06cc1d).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...

2016-05-25 Thread SparkQA
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9192#issuecomment-221774354
  
**[Test build #59341 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59341/consoleFull)**
 for PR 9192 at commit 
[`f67095e`](https://github.com/apache/spark/commit/f67095ef72540140aa2348b5262ffdf91685846a).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...

2016-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9192#issuecomment-221774407
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...

2016-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9192#issuecomment-221774409
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59341/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-15542][SparkR] Make error message clear...

2016-05-25 Thread SparkQA
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13308#issuecomment-221774053
  
**[Test build #59342 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59342/consoleFull)**
 for PR 13308 at commit 
[`cbd5163`](https://github.com/apache/spark/commit/cbd5163d73fa56a58e18598ece64aaa60e06cc1d).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [YARN][Doc][Minor] Remove several obsolete env...

2016-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13296#issuecomment-221773158
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59329/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [YARN][Doc][Minor] Remove several obsolete env...

2016-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13296#issuecomment-221773157
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [YARN][Doc][Minor] Remove several obsolete env...

2016-05-25 Thread SparkQA
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13296#issuecomment-221773071
  
**[Test build #59329 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59329/consoleFull)**
 for PR 13296 at commit 
[`367e3b8`](https://github.com/apache/spark/commit/367e3b8de0633c100bc1a9bf4742f6af80ecfa68).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-15542][SparkR] Make error message clear...

2016-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13308#issuecomment-221773031
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-15542][SparkR] Make error message clear...

2016-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13308#issuecomment-221773032
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59340/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-15542][SparkR] Make error message clear...

2016-05-25 Thread SparkQA
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13308#issuecomment-221772977
  
**[Test build #59340 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59340/consoleFull)**
 for PR 13308 at commit 
[`88319c0`](https://github.com/apache/spark/commit/88319c022b8eb55f59f8080d488e30726f475580).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-8603][SPARKR] Use shell() instead of sy...

2016-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13165#issuecomment-221772828
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...

2016-05-25 Thread shivaram
Github user shivaram commented on the pull request:

https://github.com/apache/spark/pull/9192#issuecomment-221772896
  
Thanks for the update. LGTM. Will merge after Jenkins passes. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-8603][SPARKR] Use shell() instead of sy...

2016-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13165#issuecomment-221772829
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59339/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-8603][SPARKR] Use shell() instead of sy...

2016-05-25 Thread SparkQA
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13165#issuecomment-221772778
  
**[Test build #59339 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59339/consoleFull)**
 for PR 13165 at commit 
[`0482ebb`](https://github.com/apache/spark/commit/0482ebbc43ff1bef8e7a6a16376c6ec36840a366).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...

2016-05-25 Thread asfgit
Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/13284


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...

2016-05-25 Thread shivaram
Github user shivaram commented on the pull request:

https://github.com/apache/spark/pull/13284#issuecomment-221772558
  
Yeah thats a good idea @wangmiao1981 can you open a JIRA to not mask 
`startsWith` and `endsWith` by updating our generics ? 

LGTM - Merging this to master and branch-2.0. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...

2016-05-25 Thread SparkQA
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9192#issuecomment-221772591
  
**[Test build #59341 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59341/consoleFull)**
 for PR 9192 at commit 
[`f67095e`](https://github.com/apache/spark/commit/f67095ef72540140aa2348b5262ffdf91685846a).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-15542][SparkR] Make error message clear...

2016-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13308#issuecomment-221772521
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59338/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-15542][SparkR] Make error message clear...

2016-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13308#issuecomment-221772519
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-15542][SparkR] Make error message clear...

2016-05-25 Thread SparkQA
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13308#issuecomment-221772475
  
**[Test build #59338 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59338/consoleFull)**
 for PR 13308 at commit 
[`07806de`](https://github.com/apache/spark/commit/07806de09f4be0dd9501fe81684c07a45ad68672).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...

2016-05-25 Thread felixcheung
Github user felixcheung commented on the pull request:

https://github.com/apache/spark/pull/13284#issuecomment-221772080
  
looks fine - I think we should really try to make startsWith and endsWith 
work though, but that could be a follow up.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-15542][SparkR] Make error message clear...

2016-05-25 Thread SparkQA
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13308#issuecomment-221771654
  
**[Test build #59340 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59340/consoleFull)**
 for PR 13308 at commit 
[`88319c0`](https://github.com/apache/spark/commit/88319c022b8eb55f59f8080d488e30726f475580).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-15542][SparkR] Make error message clear...

2016-05-25 Thread SparkQA
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13308#issuecomment-221771180
  
**[Test build #59338 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59338/consoleFull)**
 for PR 13308 at commit 
[`07806de`](https://github.com/apache/spark/commit/07806de09f4be0dd9501fe81684c07a45ad68672).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-8603][SPARKR] Incorrect file separator ...

2016-05-25 Thread SparkQA
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13165#issuecomment-221771183
  
**[Test build #59339 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59339/consoleFull)**
 for PR 13165 at commit 
[`0482ebb`](https://github.com/apache/spark/commit/0482ebbc43ff1bef8e7a6a16376c6ec36840a366).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-15515] [SQL] Error Handling in Running ...

2016-05-25 Thread gatorsmile
Github user gatorsmile commented on the pull request:

https://github.com/apache/spark/pull/13283#issuecomment-221770911
  
**Update**: The latest code changes contains 
- For JDBC format, we added an extra checking in the rule 
`ResolveRelations` of `Analyzer`. Without the PR, Spark will return the error 
message like: `Option 'url' not specified`. Now, we are reporting `Unsupported 
data source type for direct query on files: jdbc` 
- Make data source format name case incensitive so that error handling 
behaves consistent with the normal cases. 
- Added the test cases for all the supported formats. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-8603][SPARKR] Incorrect file separator ...

2016-05-25 Thread HyukjinKwon
Github user HyukjinKwon commented on the pull request:

https://github.com/apache/spark/pull/13165#issuecomment-221770814
  
retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-8603][SPARKR] Incorrect file separator ...

2016-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13165#issuecomment-221770536
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-8603][SPARKR] Incorrect file separator ...

2016-05-25 Thread SparkQA
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13165#issuecomment-221770524
  
**[Test build #59336 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59336/consoleFull)**
 for PR 13165 at commit 
[`0482ebb`](https://github.com/apache/spark/commit/0482ebbc43ff1bef8e7a6a16376c6ec36840a366).
 * This patch **fails MiMa tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-8603][SPARKR] Incorrect file separator ...

2016-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13165#issuecomment-221770538
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59336/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...

2016-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9192#issuecomment-221770384
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59337/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...

2016-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9192#issuecomment-221770383
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SQL] Prevent illegal NULL propagation when fi...

2016-05-25 Thread maropu
Github user maropu commented on a diff in the pull request:

https://github.com/apache/spark/pull/13290#discussion_r64688437
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
 ---
@@ -1448,6 +1450,37 @@ class Analyzer(
   }
 
   /**
+   * Fixes nullability of Attributes in a resolved LogicalPlan by using 
the nullability of
+   * corresponding Attributes of its children output Attributes. This step 
is needed because
+   * users can use a resolved AttributeReference in the Dataset API and 
outer joins
+   * can change the nullability of an AttribtueReference. Without the fix, 
a nullable column's
+   * nullable field can be actually set as non-nullable, which cause 
illegal optimization
+   * (e.g., NULL propagation) and wrong answers.
+   * See SPARK-13484 and SPARK-13801 for the concrete queries of this case.
+   */
+  object FixNullability extends Rule[LogicalPlan] {
+
+def apply(plan: LogicalPlan): LogicalPlan = plan transformUp {
+  case q: LogicalPlan if q.resolved =>
+val childrenOutput = q.children.flatMap(c => 
c.output).groupBy(_.exprId).flatMap {
--- End diff --

yes, I got your point.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...

2016-05-25 Thread SparkQA
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9192#issuecomment-221770380
  
**[Test build #59337 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59337/consoleFull)**
 for PR 9192 at commit 
[`90641a7`](https://github.com/apache/spark/commit/90641a71ff1860ddfe1a8e0bcb64cc0f0d2a56c6).
 * This patch **fails R style tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...

2016-05-25 Thread SparkQA
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9192#issuecomment-221770106
  
**[Test build #59337 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59337/consoleFull)**
 for PR 9192 at commit 
[`90641a7`](https://github.com/apache/spark/commit/90641a71ff1860ddfe1a8e0bcb64cc0f0d2a56c6).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-15534][SPARK-15535][SQL] Truncate table...

2016-05-25 Thread yhuai
Github user yhuai commented on a diff in the pull request:

https://github.com/apache/spark/pull/13302#discussion_r64688076
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala ---
@@ -288,9 +288,10 @@ case class TruncateTableCommand(
   override def run(sparkSession: SparkSession): Seq[Row] = {
 val catalog = sparkSession.sessionState.catalog
 if (!catalog.tableExists(tableName)) {
-  logError(s"table '$tableName' in TRUNCATE TABLE does not exist.")
+  throw new AnalysisException(s"Table '$tableName' in TRUNCATE TABLE 
does not exist.")
--- End diff --

Sorry Just realized that I had a typo `For drop table with IF 
EXISTS keyword` should be `For drop table without IF EXISTS keyword`.  Seems 
without IF EXISTS keyword, Hive (I tried 1.2.1) does not throw an exception if 
the table does not exist.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-8603][SPARKR] Incorrect file separator ...

2016-05-25 Thread SparkQA
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13165#issuecomment-221769561
  
**[Test build #59336 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59336/consoleFull)**
 for PR 13165 at commit 
[`0482ebb`](https://github.com/apache/spark/commit/0482ebbc43ff1bef8e7a6a16376c6ec36840a366).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-14204] [SQL] register driverClass rathe...

2016-05-25 Thread zzcclp
Github user zzcclp commented on the pull request:

https://github.com/apache/spark/pull/12000#issuecomment-221769197
  
@JoshRosen , could you merge this into branch-1.6?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-8603][SPARKR] Incorrect file separator ...

2016-05-25 Thread HyukjinKwon
Github user HyukjinKwon commented on the pull request:

https://github.com/apache/spark/pull/13165#issuecomment-221769185
  
I just addressed the comments and here is the 
(stdout)[https://gist.github.com/HyukjinKwon/4bf35184f3a30f3bce987a58ec2bbbab] 
of testing on Windiws.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-15534][SPARK-15535][SQL] Truncate table...

2016-05-25 Thread tejasapatil
Github user tejasapatil commented on a diff in the pull request:

https://github.com/apache/spark/pull/13302#discussion_r64687912
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala ---
@@ -288,9 +288,10 @@ case class TruncateTableCommand(
   override def run(sparkSession: SparkSession): Seq[Row] = {
 val catalog = sparkSession.sessionState.catalog
 if (!catalog.tableExists(tableName)) {
-  logError(s"table '$tableName' in TRUNCATE TABLE does not exist.")
+  throw new AnalysisException(s"Table '$tableName' in TRUNCATE TABLE 
does not exist.")
--- End diff --

@yhuai : Hive is doing the right thing of not throwing an exception when 
`IF EXISTS` keyword is used in `DROP TABLE` (irrespective of the table being 
there or not). If end user expects exception to be thrown, then one should not 
use the `IF EXISTS` in the command. Having `IF EXISTS` helps for writing 
pipelines which run daily and one wants "drop if exists or do nothing" 
semantics.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...

2016-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9192#issuecomment-221768776
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...

2016-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9192#issuecomment-221768777
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59335/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...

2016-05-25 Thread SparkQA
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9192#issuecomment-221768774
  
**[Test build #59335 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59335/consoleFull)**
 for PR 9192 at commit 
[`640ffca`](https://github.com/apache/spark/commit/640ffcaa2836d40337cda6c3ba1e51f30d16a44c).
 * This patch **fails R style tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SQL] Prevent illegal NULL propagation when fi...

2016-05-25 Thread SparkQA
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13290#issuecomment-221768479
  
**[Test build #59334 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59334/consoleFull)**
 for PR 13290 at commit 
[`127024d`](https://github.com/apache/spark/commit/127024da7e1058cd39b71e85c6dcd08b5e3e2b53).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...

2016-05-25 Thread SparkQA
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9192#issuecomment-221768460
  
**[Test build #59335 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59335/consoleFull)**
 for PR 9192 at commit 
[`640ffca`](https://github.com/apache/spark/commit/640ffcaa2836d40337cda6c3ba1e51f30d16a44c).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-15532] [SQL] Add SQLConf.ALLOW_MULTIPLE...

2016-05-25 Thread SparkQA
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13310#issuecomment-221768478
  
**[Test build #59333 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59333/consoleFull)**
 for PR 13310 at commit 
[`f40a898`](https://github.com/apache/spark/commit/f40a89873ba92eaf5821dce4728d2aab84e1289e).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SQL] Prevent illegal NULL propagation when fi...

2016-05-25 Thread yhuai
Github user yhuai commented on a diff in the pull request:

https://github.com/apache/spark/pull/13290#discussion_r64687394
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
 ---
@@ -1448,6 +1450,37 @@ class Analyzer(
   }
 
   /**
+   * Fixes nullability of Attributes in a resolved LogicalPlan by using 
the nullability of
+   * corresponding Attributes of its children output Attributes. This step 
is needed because
+   * users can use a resolved AttributeReference in the Dataset API and 
outer joins
+   * can change the nullability of an AttribtueReference. Without the fix, 
a nullable column's
+   * nullable field can be actually set as non-nullable, which cause 
illegal optimization
+   * (e.g., NULL propagation) and wrong answers.
+   * See SPARK-13484 and SPARK-13801 for the concrete queries of this case.
+   */
+  object FixNullability extends Rule[LogicalPlan] {
+
+def apply(plan: LogicalPlan): LogicalPlan = plan transformUp {
+  case q: LogicalPlan if q.resolved =>
+val childrenOutput = q.children.flatMap(c => 
c.output).groupBy(_.exprId).flatMap {
+  case (exprId, attributes) =>
+// If there are multiple Attributes having the same ExpirId, 
we need to resolve
+// the conflict of nullable field.
+val nullable = attributes.map(_.nullable).reduce(_ || _)
--- End diff --

I feel it is not very possible. Let me think about it more.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-15538][SPARK-15539][SQL] Truncate table...

2016-05-25 Thread sureshthalamati
Github user sureshthalamati commented on the pull request:

https://github.com/apache/spark/pull/13315#issuecomment-221768017
  
@andrewor14 
I tried  this PR.  It  does not seem to work/raise error if user attempts 
to truncate on partition table  with out specifying the partition spec.
val df = Seq((1 , "john", "CA") ,(2,"Mike", "NY"), (3, "Robert", 
"CA")).toDF("id", "name", "state")
df.write.format("parquet").partitionBy("state").saveAsTable("emp16")
sql("truncate table emp16") 
sql("select count(*) from emp16").show




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...

2016-05-25 Thread felixcheung
Github user felixcheung commented on a diff in the pull request:

https://github.com/apache/spark/pull/9192#discussion_r64687334
  
--- Diff: R/pkg/inst/tests/testthat/test_sparkSQL.R ---
@@ -169,48 +169,50 @@ test_that("create DataFrame from RDD", {
   error = function(err) {
 skip("Hive is not build with SparkSQL, skipped")
   })
-  sql(hiveCtx, "CREATE TABLE people (name string, age double, height 
float)")
-  df <- read.df(hiveCtx, jsonPathNa, "json", schema)
+  assign(".sparkRHivesc", hiveCtx, envir = .sparkREnv)
--- End diff --

I'll refactor this.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-15515] [SQL] Error Handling in Running ...

2016-05-25 Thread SparkQA
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13283#issuecomment-221767896
  
**[Test build #59332 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59332/consoleFull)**
 for PR 13283 at commit 
[`76f4f80`](https://github.com/apache/spark/commit/76f4f80f962e0271a2073a4cb8de0d513013cf87).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...

2016-05-25 Thread felixcheung
Github user felixcheung commented on the pull request:

https://github.com/apache/spark/pull/9192#issuecomment-221767863
  
@shivaram it's true there's some scaffolding we need to add (though I'm 
pretty we could codegen them on the fly instead). I think the idea is this is 
temporary and in the next release (2.1.0?) we could remove these very easily 
(just a few lines before and after the method, plus renaming x.default back to 
x) to stop supporting deprecated calls.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...

2016-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9192#issuecomment-221767615
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...

2016-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9192#issuecomment-221767617
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59331/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...

2016-05-25 Thread SparkQA
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9192#issuecomment-221767613
  
**[Test build #59331 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59331/consoleFull)**
 for PR 9192 at commit 
[`98e7ab9`](https://github.com/apache/spark/commit/98e7ab978dd1a7aba06ee47eb012c496f00cbe3b).
 * This patch **fails R style tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...

2016-05-25 Thread SparkQA
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9192#issuecomment-221767353
  
**[Test build #59331 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59331/consoleFull)**
 for PR 9192 at commit 
[`98e7ab9`](https://github.com/apache/spark/commit/98e7ab978dd1a7aba06ee47eb012c496f00cbe3b).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...

2016-05-25 Thread felixcheung
Github user felixcheung commented on a diff in the pull request:

https://github.com/apache/spark/pull/9192#discussion_r64686844
  
--- Diff: R/pkg/R/SQLContext.R ---
@@ -362,12 +433,14 @@ sql <- function(sqlContext, sqlQuery) {
 #' sc <- sparkR.init()
 #' sqlContext <- sparkRSQL.init(sc)
 #' path <- "path/to/file.json"
-#' df <- read.json(sqlContext, path)
+#' df <- read.json(path)
 #' registerTempTable(df, "table")
-#' new_df <- tableToDF(sqlContext, "table")
+#' new_df <- tableToDF("table")
 #' }
+#' @note since 2.0.0
--- End diff --

not yet, I don;t want to keep checking it, that would be the next PR.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-15532] [SQL] Add SQLConf.ALLOW_MULTIPLE...

2016-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13310#issuecomment-221766492
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-15532] [SQL] Add SQLConf.ALLOW_MULTIPLE...

2016-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13310#issuecomment-221766493
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59330/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-15532] [SQL] Add SQLConf.ALLOW_MULTIPLE...

2016-05-25 Thread SparkQA
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13310#issuecomment-221766483
  
**[Test build #59330 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59330/consoleFull)**
 for PR 13310 at commit 
[`bed21b0`](https://github.com/apache/spark/commit/bed21b01beecae4e8737ffef0630f5f9889c735f).
 * This patch **fails MiMa tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...

2016-05-25 Thread sun-rui
Github user sun-rui commented on the pull request:

https://github.com/apache/spark/pull/9192#issuecomment-221766378
  
LGTM except some minor comments.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...

2016-05-25 Thread felixcheung
Github user felixcheung commented on a diff in the pull request:

https://github.com/apache/spark/pull/9192#discussion_r64686315
  
--- Diff: R/pkg/R/SQLContext.R ---
@@ -254,6 +301,7 @@ jsonFile <- function(sqlContext, path) {
 #' df <- jsonRDD(sqlContext, rdd)
 #'}
 
+# TODO: remove - this method is no longer exported
--- End diff --

opened SPARK-15545
one reason to not remove this right away is because we have been talking 
about calling internal methods via `SparkR:::jsonRDD` for this and other RDD 
methods.
we have .Deprecated() here though for a warning


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SQL] Prevent illegal NULL propagation when fi...

2016-05-25 Thread yhuai
Github user yhuai commented on a diff in the pull request:

https://github.com/apache/spark/pull/13290#discussion_r64686215
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
 ---
@@ -1448,6 +1450,37 @@ class Analyzer(
   }
 
   /**
+   * Fixes nullability of Attributes in a resolved LogicalPlan by using 
the nullability of
+   * corresponding Attributes of its children output Attributes. This step 
is needed because
+   * users can use a resolved AttributeReference in the Dataset API and 
outer joins
+   * can change the nullability of an AttribtueReference. Without the fix, 
a nullable column's
+   * nullable field can be actually set as non-nullable, which cause 
illegal optimization
+   * (e.g., NULL propagation) and wrong answers.
+   * See SPARK-13484 and SPARK-13801 for the concrete queries of this case.
+   */
+  object FixNullability extends Rule[LogicalPlan] {
+
+def apply(plan: LogicalPlan): LogicalPlan = plan transformUp {
+  case q: LogicalPlan if q.resolved =>
+val childrenOutput = q.children.flatMap(c => 
c.output).groupBy(_.exprId).flatMap {
--- End diff --

Yea. This version tries to fix the nullability for the entire query plan 
tree because I think it is possible to hit this problem in other cases when 
using Dataset API.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...

2016-05-25 Thread sun-rui
Github user sun-rui commented on a diff in the pull request:

https://github.com/apache/spark/pull/9192#discussion_r64686229
  
--- Diff: R/pkg/R/jobj.R ---
@@ -77,6 +77,11 @@ print.jobj <- function(x, ...) {
   cat("Java ref type", name, "id", x$id, "\n", sep = " ")
 }
 
+getClassName.jobj <- function(x) {
--- End diff --

Could you address this comment?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-15532] [SQL] Add SQLConf.ALLOW_MULTIPLE...

2016-05-25 Thread SparkQA
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13310#issuecomment-221765755
  
**[Test build #59330 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59330/consoleFull)**
 for PR 13310 at commit 
[`bed21b0`](https://github.com/apache/spark/commit/bed21b01beecae4e8737ffef0630f5f9889c735f).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...

2016-05-25 Thread felixcheung
Github user felixcheung commented on a diff in the pull request:

https://github.com/apache/spark/pull/9192#discussion_r64686152
  
--- Diff: R/pkg/R/SQLContext.R ---
@@ -37,6 +37,37 @@ getInternalType <- function(x) {
  stop(paste("Unsupported type for SparkDataFrame:", class(x
 }
 
+#' Temporary function to reroute old S3 Method call to new
+#' We need to check the class of x to ensure it is SQLContext before 
dispatching
+dispatchFunc <- function(newFuncSig, x, ...) {
--- End diff --

I think this is very specific to this file - this helper is implemented to 
specifically check & remove sqlContext parameter; I'll add more documentation 
on this.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-15543][SQL] Rename DefaultSources to ma...

2016-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13311#issuecomment-221765444
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59326/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-15543][SQL] Rename DefaultSources to ma...

2016-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13311#issuecomment-221765440
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-15543][SQL] Rename DefaultSources to ma...

2016-05-25 Thread SparkQA
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13311#issuecomment-221765335
  
**[Test build #59327 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59327/consoleFull)**
 for PR 13311 at commit 
[`51dcde9`](https://github.com/apache/spark/commit/51dcde9b1274b62fbd0a29c8cdfcb68d70a65f76).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-15543][SQL] Rename DefaultSources to ma...

2016-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13311#issuecomment-221765416
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59327/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-15543][SQL] Rename DefaultSources to ma...

2016-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13311#issuecomment-221765415
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-15543][SQL] Rename DefaultSources to ma...

2016-05-25 Thread SparkQA
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13311#issuecomment-221765364
  
**[Test build #59326 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59326/consoleFull)**
 for PR 13311 at commit 
[`eed27f0`](https://github.com/apache/spark/commit/eed27f035b1ffee9a64a1276e3fd34a0e8fe27f8).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [Spark-15129][R][DOC][WIP]R API changes in ML

2016-05-25 Thread GayathriMurali
Github user GayathriMurali commented on the pull request:

https://github.com/apache/spark/pull/13285#issuecomment-221764817
  
@yanboliang Thanks, thats a good idea. However, that would just include 
example code and not how the output of summary() looks like. It might be useful 
to include that


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-14634][ML] Add BisectingKMeansSummary

2016-05-25 Thread zhengruifeng
Github user zhengruifeng commented on the pull request:

https://github.com/apache/spark/pull/12394#issuecomment-221764870
  
cc @mengxr 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...

2016-05-25 Thread sun-rui
Github user sun-rui commented on a diff in the pull request:

https://github.com/apache/spark/pull/9192#discussion_r64685560
  
--- Diff: R/pkg/R/SQLContext.R ---
@@ -362,12 +433,14 @@ sql <- function(sqlContext, sqlQuery) {
 #' sc <- sparkR.init()
 #' sqlContext <- sparkRSQL.init(sc)
 #' path <- "path/to/file.json"
-#' df <- read.json(sqlContext, path)
+#' df <- read.json(path)
 #' registerTempTable(df, "table")
-#' new_df <- tableToDF(sqlContext, "table")
+#' new_df <- tableToDF("table")
 #' }
+#' @note since 2.0.0
--- End diff --

remove?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...

2016-05-25 Thread sun-rui
Github user sun-rui commented on a diff in the pull request:

https://github.com/apache/spark/pull/9192#discussion_r64685428
  
--- Diff: R/pkg/R/SQLContext.R ---
@@ -362,12 +433,14 @@ sql <- function(sqlContext, sqlQuery) {
 #' sc <- sparkR.init()
 #' sqlContext <- sparkRSQL.init(sc)
 #' path <- "path/to/file.json"
-#' df <- read.json(sqlContext, path)
+#' df <- read.json(path)
 #' registerTempTable(df, "table")
-#' new_df <- tableToDF(sqlContext, "table")
+#' new_df <- tableToDF("table")
 #' }
+#' @note since 2.0.0
--- End diff --

?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-15538][SQL] Adding error check for trun...

2016-05-25 Thread sureshthalamati
Github user sureshthalamati commented on the pull request:

https://github.com/apache/spark/pull/13305#issuecomment-221762895
  
Thank you Andrew , Herman for your input. I am  closing this pull request. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-15538][SQL] Adding error check for trun...

2016-05-25 Thread sureshthalamati
Github user sureshthalamati closed the pull request at:

https://github.com/apache/spark/pull/13305


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [YARN][Doc][Minor] Remove several obsolete env...

2016-05-25 Thread SparkQA
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13296#issuecomment-221761576
  
**[Test build #59329 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59329/consoleFull)**
 for PR 13296 at commit 
[`367e3b8`](https://github.com/apache/spark/commit/367e3b8de0633c100bc1a9bf4742f6af80ecfa68).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-15100][DOC] Modified user guide and exa...

2016-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13176#issuecomment-221761348
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



  1   2   3   4   5   6   7   >