[GitHub] spark pull request: [SPARK-5297][Streaming][backport] Backport SPA...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4154#issuecomment-70983713
  
  [Test build #25954 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25954/consoleFull)
 for   PR 4154 at commit 
[`a4b2bea`](https://github.com/apache/spark/commit/a4b2bea998e5722b11383d553914253249788c2c).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3726] [MLlib] Allow sampling_rate not e...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4073#issuecomment-70983710
  
  [Test build #25955 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25955/consoleFull)
 for   PR 4073 at commit 
[`d5d68e7`](https://github.com/apache/spark/commit/d5d68e79f7ccbefe4c45d53253df3e7066cb7f53).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5196][SQL] Support `comment` in Create ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/3999#issuecomment-70983364
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25949/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5196][SQL] Support `comment` in Create ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/3999#issuecomment-70983356
  
  [Test build #25949 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25949/consoleFull)
 for   PR 3999 at commit 
[`1c71505`](https://github.com/apache/spark/commit/1c71505c5b44cf4fd75fb7e18ec8e43987d11f32).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3726] [MLlib] Allow sampling_rate not e...

[MechCoder]

Github user MechCoder commented on the pull request:

https://github.com/apache/spark/pull/4073#issuecomment-70983304
  
Repushed after fixing the style checks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5297][Streaming][backport] Backport SPA...

[jerryshao]

Github user jerryshao commented on the pull request:

https://github.com/apache/spark/pull/4154#issuecomment-70983328
  
Jenkins, retest this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5297][Streaming][backport] Backport SPA...

[jerryshao]

Github user jerryshao commented on the pull request:

https://github.com/apache/spark/pull/4154#issuecomment-70983308
  
Quite strange the test is OK in my local test but failed in Jenkins.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3726] [MLlib] Allow sampling_rate not e...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4073#issuecomment-70982966
  
  [Test build #25953 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25953/consoleFull)
 for   PR 4073 at commit 
[`8a0acb5`](https://github.com/apache/spark/commit/8a0acb585fd2c54562d30628e384404d8f744690).
 * This patch **fails Scala style tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3726] [MLlib] Allow sampling_rate not e...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/4073#issuecomment-70982967
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25953/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3726] [MLlib] Allow sampling_rate not e...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4073#issuecomment-70982888
  
  [Test build #25953 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25953/consoleFull)
 for   PR 4073 at commit 
[`8a0acb5`](https://github.com/apache/spark/commit/8a0acb585fd2c54562d30628e384404d8f744690).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5147][Streaming] Delete the received da...

[tdas]

Github user tdas commented on the pull request:

https://github.com/apache/spark/pull/4037#issuecomment-70982900
  
Mind closing this PR?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3726] [MLlib] Allow sampling_rate not e...

[MechCoder]

Github user MechCoder commented on the pull request:

https://github.com/apache/spark/pull/4073#issuecomment-70982719
  
@jkbradley Thanks for the tip. Fixed. Anything more?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...

[OopsOutOfMemory]

Github user OopsOutOfMemory commented on a diff in the pull request:

https://github.com/apache/spark/pull/4127#discussion_r23359441
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/commands.scala ---
@@ -178,3 +180,34 @@ case class DescribeCommand(
 child.output.map(field => Row(field.name, field.dataType.toString, 
null))
   }
 }
+
+/**
+ * :: DeveloperApi ::
+ */
+@DeveloperApi
+case class DDLDescribeCommand(
+dbName: Option[String],
+tableName: String, isExtended: Boolean) extends RunnableCommand {
+
+  override def run(sqlContext: SQLContext) = {
+val tblRelation = dbName match {
+  case Some(db) => UnresolvedRelation(Seq(db, tableName))
+  case None => UnresolvedRelation(Seq(tableName))
+}
+val logicalRelation = sqlContext.executePlan(tblRelation).analyzed
+val rows = new ArrayBuffer[Row]()
+rows ++= logicalRelation.schema.fields.map{field =>
+  Row(field.name, field.dataType.toSimpleString, null)}
+
+/*
+ * TODO if future support partition table, add header below:
+ * # Partition Information
+ * # col_name data_type comment
--- End diff --

@marmbrus yeah, I understand your concern.
I refereed hive's describe implementation. 
We should provide a  `getPartitionedColumns` method in partition table. 
Maybe the partition info description can be done after 
[SPARK-5182](https://issues.apache.org/jira/browse/SPARK-5182). 
What about just make the PR only do `describe table`. After 
[SPARK-5182](https://issues.apache.org/jira/browse/SPARK-5182) , we can do 
`extended` operation and show partition columns info ? /cc @rxin 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5147][Streaming] Delete the received da...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/4149


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5147][Streaming] Delete the received da...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4149#issuecomment-70982500
  
  [Test build #25952 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25952/consoleFull)
 for   PR 4149 at commit 
[`730798b`](https://github.com/apache/spark/commit/730798b8a21eea5ebc13acb9e4376b42271ff9e4).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5147][Streaming] Delete the received da...

[tdas]

Github user tdas commented on the pull request:

https://github.com/apache/spark/pull/4149#issuecomment-70982298
  
@pwendell Since I made changes only to the comments I am merging this.
@jerryshao I will take a look at that PR.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5118][SQL] Fix: create table test store...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/3921#issuecomment-70982063
  
  [Test build #25948 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25948/consoleFull)
 for   PR 3921 at commit 
[`b1ba3be`](https://github.com/apache/spark/commit/b1ba3be2df7f3681d65b1beb07284a3dcf8f267d).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5118][SQL] Fix: create table test store...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/3921#issuecomment-70982072
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25948/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5213] [SQL] Sql Parser dialect support

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4015#issuecomment-70981618
  
  [Test build #25947 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25947/consoleFull)
 for   PR 4015 at commit 
[`b0e8084`](https://github.com/apache/spark/commit/b0e8084524ae7d33f295e8caae4e726dfcd5be3e).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `  protected case class Keyword(str: String) `
  * `class SqlLexical extends StdLexical `
  * `abstract class SQLDialect `
  * `class DefaultSQLDialect extends SQLDialect `
  * `class HiveQLDialect extends SQLDialect `



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5213] [SQL] Sql Parser dialect support

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/4015#issuecomment-70981624
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25947/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SQL] SPARK-5309: Use Dictionary for Binary->S...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4139#issuecomment-70980598
  
  [Test build #25946 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25946/consoleFull)
 for   PR 4139 at commit 
[`0af7c41`](https://github.com/apache/spark/commit/0af7c41e129afb630b351dea5f409ddc0c6d01ff).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5297][Streaming][backport] Backport SPA...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4154#issuecomment-70980557
  
  [Test build #25945 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25945/consoleFull)
 for   PR 4154 at commit 
[`a4b2bea`](https://github.com/apache/spark/commit/a4b2bea998e5722b11383d553914253249788c2c).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SQL] SPARK-5309: Use Dictionary for Binary->S...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/4139#issuecomment-70980603
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25946/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5297][Streaming][backport] Backport SPA...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/4154#issuecomment-70980558
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25945/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5154] [PySpark] [Streaming] Kafka strea...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/3715#issuecomment-70980275
  
  [Test build #25951 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25951/consoleFull)
 for   PR 3715 at commit 
[`adeeb38`](https://github.com/apache/spark/commit/adeeb3863353f9a0ca3070a9cc914a2914d95fa9).
 * This patch **does not merge cleanly**.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5317]Set BoostingStrategy.defaultParams...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/4103


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5196][SQL] Support `comment` in Create ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/3999#issuecomment-70979284
  
  [Test build #25950 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25950/consoleFull)
 for   PR 3999 at commit 
[`d1cfb0f`](https://github.com/apache/spark/commit/d1cfb0fa7620c3cc6800087f32a2e4542884823f).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5196][SQL] Support `comment` in Create ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/3999#issuecomment-70978608
  
  [Test build #25949 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25949/consoleFull)
 for   PR 3999 at commit 
[`1c71505`](https://github.com/apache/spark/commit/1c71505c5b44cf4fd75fb7e18ec8e43987d11f32).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

[WangTaoTheTonic]

Github user WangTaoTheTonic commented on a diff in the pull request:

https://github.com/apache/spark/pull/4129#discussion_r23357881
  
--- Diff: 
core/src/main/scala/org/apache/spark/scheduler/cluster/SparkDeploySchedulerBackend.scala
 ---
@@ -40,6 +40,14 @@ private[spark] class SparkDeploySchedulerBackend(
   var registrationDone = false
 
   val maxCores = conf.getOption("spark.cores.max").map(_.toInt)
+  val coreNumPerTask = {
+val corePerTask = conf.getInt("spark.task.cpus", 1)
+if (corePerTask < 1) {
+  throw new IllegalArgumentException(
+s"spark.task.cpus is set to an invalid value $corePerTask")
+}
+corePerTask
+  }
--- End diff --

Is `corePerTask ` necessary? Why don't just assign 
`conf.getInt("spark.task.cpus", 1)` to `coreNumPerTask `? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5326] Show fetch wait time as optional ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4110#issuecomment-70978216
  
  [Test build #25944 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25944/consoleFull)
 for   PR 4110 at commit 
[`610051e`](https://github.com/apache/spark/commit/610051e76cc3405ad0cffb2295126cea0b013491).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5326] Show fetch wait time as optional ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/4110#issuecomment-70978222
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25944/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5118][SQL] Fix: create table test store...

[guowei2]

Github user guowei2 commented on the pull request:

https://github.com/apache/spark/pull/3921#issuecomment-70977693
  
@yhuai  I use `DESC EXTENDED` to check the the stored file, Is it OK?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5262] [SQL] coalesce should allow NullT...

[yhuai]

Github user yhuai commented on the pull request:

https://github.com/apache/spark/pull/4057#issuecomment-70977695
  
Although coalesce only returns the first non-null value, but we need to 
correctly set the type of coalesce. Let's say we have two columns. The type of 
one is LongType and the type of another one is IntType, when we do coalesce, 
the return type of coalesce should be LongType. Right now, there will be an 
error when we are trying to call coalesce for these two columns. In 
FunctionArgumentConversion, we should make sure all expressions of coalesce are 
in the same type and if they are not, we should add cast.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5118][SQL] Fix: create table test store...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/3921#issuecomment-70977566
  
  [Test build #25948 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25948/consoleFull)
 for   PR 3921 at commit 
[`b1ba3be`](https://github.com/apache/spark/commit/b1ba3be2df7f3681d65b1beb07284a3dcf8f267d).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5213] [SQL] Sql Parser dialect support

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4015#issuecomment-70977183
  
  [Test build #25947 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25947/consoleFull)
 for   PR 4015 at commit 
[`b0e8084`](https://github.com/apache/spark/commit/b0e8084524ae7d33f295e8caae4e726dfcd5be3e).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4786][SQL]: Parquet filter pushdown for...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/4156#issuecomment-70976504
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4786][SQL]: Parquet filter pushdown for...

[saucam]

Github user saucam commented on the pull request:

https://github.com/apache/spark/pull/4156#issuecomment-70976498
  
done


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5317]Set BoostingStrategy.defaultParams...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/4103#issuecomment-70976404
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25943/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5317]Set BoostingStrategy.defaultParams...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4103#issuecomment-70976399
  
  [Test build #25943 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25943/consoleFull)
 for   PR 4103 at commit 
[`87bab1c`](https://github.com/apache/spark/commit/87bab1c427f917a757a30af56288f010d16a4107).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-4786: Parquet filter pushdown for castab...

[sryza]

Github user sryza commented on the pull request:

https://github.com/apache/spark/pull/4156#issuecomment-70976336
  
Hi @saucam, mind tagging this with [SQL] so it can get properly sorted?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-4786: Parquet filter pushdown for castab...

[saucam]

GitHub user saucam opened a pull request:

https://github.com/apache/spark/pull/4156

SPARK-4786: Parquet filter pushdown for castable types

Enable parquet filter pushdown of castable types like short, byte that can 
be cast to integer

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/saucam/spark filter_short

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/4156.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4156


commit cb2e0d94102bedb961f403bdc2420fabc021fe1a
Author: Yash Datta 
Date:   2015-01-22T06:00:18Z

SPARK-4786: Parquet filter pushdown for castable types




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SQL] SPARK-5309: Use Dictionary for Binary->S...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4139#issuecomment-70976187
  
  [Test build #25946 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25946/consoleFull)
 for   PR 4139 at commit 
[`0af7c41`](https://github.com/apache/spark/commit/0af7c41e129afb630b351dea5f409ddc0c6d01ff).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SQL] SPARK-5309: Use Dictionary for Binary->S...

[ash211]

Github user ash211 commented on the pull request:

https://github.com/apache/spark/pull/4139#issuecomment-70975989
  
Jenkins this is ok to test


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5009] [SQL] Long keyword support in SQL...

[chenghao-intel]

Github user chenghao-intel commented on a diff in the pull request:

https://github.com/apache/spark/pull/3926#discussion_r23357031
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/AbstractSparkSQLParser.scala
 ---
@@ -25,15 +25,42 @@ import scala.util.parsing.input.CharArrayReader.EofCh
 
 import org.apache.spark.sql.catalyst.plans.logical._
 
+private[sql] object KeywordNormalizer {
--- End diff --

It's also used within`SqlLexical.processIdent`, but you're right, we'd 
better keep it the minimize visibility. I will do that in #4015 .


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5297][Streaming][backport] Backport SPA...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4154#issuecomment-70974245
  
  [Test build #25945 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25945/consoleFull)
 for   PR 4154 at commit 
[`a4b2bea`](https://github.com/apache/spark/commit/a4b2bea998e5722b11383d553914253249788c2c).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5326] Show fetch wait time as optional ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4110#issuecomment-70973678
  
  [Test build #25944 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25944/consoleFull)
 for   PR 4110 at commit 
[`610051e`](https://github.com/apache/spark/commit/610051e76cc3405ad0cffb2295126cea0b013491).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5326] Show fetch wait time as optional ...

[kayousterhout]

Github user kayousterhout commented on the pull request:

https://github.com/apache/spark/pull/4110#issuecomment-70973485
  
Thanks for the thorough look @JoshRosen -- I'll merge this once it passes 
tests (if there are no more comments)!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5317]Set BoostingStrategy.defaultParams...

[mengxr]

Github user mengxr commented on the pull request:

https://github.com/apache/spark/pull/4103#issuecomment-70973213
  
LGTM pending Jenkins.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3424][MLLIB] cache point distances duri...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/4144


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3424][MLLIB] cache point distances duri...

[mengxr]

Github user mengxr commented on the pull request:

https://github.com/apache/spark/pull/4144#issuecomment-70973105
  
I've merged this into master.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3424][MLLIB] cache point distances duri...

[mengxr]

Github user mengxr commented on the pull request:

https://github.com/apache/spark/pull/4144#issuecomment-70972911
  
Sure. Thanks for keeping your implementation updated!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5317]Set BoostingStrategy.defaultParams...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4103#issuecomment-70972189
  
  [Test build #25943 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25943/consoleFull)
 for   PR 4103 at commit 
[`87bab1c`](https://github.com/apache/spark/commit/87bab1c427f917a757a30af56288f010d16a4107).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: fix for Spark-5335: Destroying cluster in VPC ...

[nchammas]

Github user nchammas commented on the pull request:

https://github.com/apache/spark/pull/4122#issuecomment-70971228
  
I guess review on this can wait until #4038 is merged. You will probably 
need to rebase this PR at that time.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5297][Streaming][backport] Backport SPA...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4154#issuecomment-70970274
  
  [Test build #25942 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25942/consoleFull)
 for   PR 4154 at commit 
[`a91f295`](https://github.com/apache/spark/commit/a91f295abee7aaf758c6c9829ab62e36ca0b0022).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5297][Streaming][backport] Backport SPA...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/4154#issuecomment-70970280
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25942/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4879] Use the Spark driver to authorize...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4155#issuecomment-70969881
  
**[Test build #25941 timed 
out](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25941/consoleFull)**
 for PR 4155 at commit 
[`1c2b219`](https://github.com/apache/spark/commit/1c2b21908225a644f268adb8e1e67a4479492532)
 after a configured wait of `120m`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4879] Use the Spark driver to authorize...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/4155#issuecomment-70969885
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25941/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3424][MLLIB] cache point distances duri...

[derrickburns]

Github user derrickburns commented on the pull request:

https://github.com/apache/spark/pull/4144#issuecomment-70969423
  
@mengxr
You may want to refer to the newer code in GitHub
 instead of
the old PR going forward.

I've refactored the PointOps interface since the old PR following your
comments.

I also rewrote the KMeansPlusPlus in a similar fashion as to your changes
to KMeansParallel and support sparse points in high dimension, but the
PointOps redesign is the most important change.

On Wed, Jan 21, 2015 at 6:16 PM, UCB AMPLab 
wrote:

> Test PASSed.
> Refer to this link for build results (access rights to CI server needed):
> https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25938/
> Test PASSed.
>
> —
> Reply to this email directly or view it on GitHub
> .
>


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4879] Use the Spark driver to authorize...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/4155#issuecomment-70968413
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25940/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4879] Use the Spark driver to authorize...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4155#issuecomment-70968409
  
**[Test build #25940 timed 
out](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25940/consoleFull)**
 for PR 4155 at commit 
[`6b543ba`](https://github.com/apache/spark/commit/6b543bae5d9a6ea27408a57663986d3947ea7ab1)
 after a configured wait of `120m`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5297][Streaming][backport] Backport SPA...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4154#issuecomment-70965488
  
  [Test build #25942 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25942/consoleFull)
 for   PR 4154 at commit 
[`a91f295`](https://github.com/apache/spark/commit/a91f295abee7aaf758c6c9829ab62e36ca0b0022).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5297][Streaming][backport] Backport SPA...

[jerryshao]

Github user jerryshao commented on the pull request:

https://github.com/apache/spark/pull/4154#issuecomment-70965187
  
Jenkins, retest this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3562]Periodic cleanup event logs

[suyanNone]

Github user suyanNone commented on the pull request:

https://github.com/apache/spark/pull/2471#issuecomment-70964880
  
Is thi patch ok to merge?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5297][Streaming][backport] Backport SPA...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/4154#issuecomment-70964374
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25939/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5297][Streaming][backport] Backport SPA...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4154#issuecomment-70964370
  
  [Test build #25939 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25939/consoleFull)
 for   PR 4154 at commit 
[`a91f295`](https://github.com/apache/spark/commit/a91f295abee7aaf758c6c9829ab62e36ca0b0022).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

[CodingCat]

Github user CodingCat commented on a diff in the pull request:

https://github.com/apache/spark/pull/4129#discussion_r23352167
  
--- Diff: 
core/src/main/scala/org/apache/spark/scheduler/cluster/SparkDeploySchedulerBackend.scala
 ---
@@ -40,6 +40,14 @@ private[spark] class SparkDeploySchedulerBackend(
   var registrationDone = false
 
   val maxCores = conf.getOption("spark.cores.max").map(_.toInt)
+  val coreNumPerTask = {
+val corePerTask = conf.getInt("spark.task.cpus", 1)
+if (corePerTask < 1) {
+  throw new IllegalArgumentException(
+s"spark.task.cpus is set to an invalid value $corePerTask")
+}
+corePerTask
+  }
--- End diff --

ah? what do you mean?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5147][Streaming] Delete the received da...

[jerryshao]

Github user jerryshao commented on the pull request:

https://github.com/apache/spark/pull/4149#issuecomment-70962102
  
Thanks @tdas for your refactoring, I think we should fix this #4032 
accordingly, otherwise we will still meet such kind of error when recovering 
from failure according to my test of your branch:

```
Exception in thread "main" org.apache.spark.SparkException: Job aborted due 
to stage failure: Task 0 in stage 73.0 failed 1 times, most recent failure: 
Lost task 0.0 in stage 73.0 (TID 166, localhost): 
java.io.FileNotFoundException: 
file:/home/jerryshao/project/apache-spark/checkpoint-wal-test/receivedData/0/log-1421893625401-1421893685401
at 
org.apache.hadoop.fs.RawLocalFileSystem.open(RawLocalFileSystem.java:175)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:427)
at 
org.apache.spark.streaming.util.HdfsUtils$.getInputStream(HdfsUtils.scala:45)
at 
org.apache.spark.streaming.util.WriteAheadLogRandomReader.(WriteAheadLogRandomReader.scala:32)
at 
org.apache.spark.streaming.rdd.WriteAheadLogBackedBlockRDD.compute(WriteAheadLogBackedBlockRDD.scala:99)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:264)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:231)
at 
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:264)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:231)
at 
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:264)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:231)
at 
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:264)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:231)
at 
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
at 
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
at org.apache.spark.scheduler.Task.run(Task.scala:56)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:183)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4879] Use the Spark driver to authorize...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4155#issuecomment-70961725
  
  [Test build #25941 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25941/consoleFull)
 for   PR 4155 at commit 
[`1c2b219`](https://github.com/apache/spark/commit/1c2b21908225a644f268adb8e1e67a4479492532).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

[WangTaoTheTonic]

Github user WangTaoTheTonic commented on a diff in the pull request:

https://github.com/apache/spark/pull/4129#discussion_r23351906
  
--- Diff: 
core/src/main/scala/org/apache/spark/scheduler/cluster/SparkDeploySchedulerBackend.scala
 ---
@@ -40,6 +40,14 @@ private[spark] class SparkDeploySchedulerBackend(
   var registrationDone = false
 
   val maxCores = conf.getOption("spark.cores.max").map(_.toInt)
+  val coreNumPerTask = {
+val corePerTask = conf.getInt("spark.task.cpus", 1)
+if (corePerTask < 1) {
+  throw new IllegalArgumentException(
+s"spark.task.cpus is set to an invalid value $corePerTask")
+}
+corePerTask
+  }
--- End diff --

Seems `corePerTask` and  `coreNumPerTask` have some redundancies.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5262] [SQL] coalesce should allow NullT...

[adrian-wang]

Github user adrian-wang commented on the pull request:

https://github.com/apache/spark/pull/4057#issuecomment-70961348
  
@yhuai I don't think we need to cast data types, since coalesce() only 
return the first non-null value from its parameters, so use the first 
parameter's datatype is OK. I was adding tests in ExpressionEvaluationSuite but 
was blocked by some code-gen tests, and I am working on that.

@marmbrus it seems #4040 is reverted due to some conflicts... should I port 
all the code into this PR together?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3424][MLLIB] cache point distances duri...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/4144#issuecomment-70960725
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25938/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3424][MLLIB] cache point distances duri...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4144#issuecomment-70960719
  
  [Test build #25938 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25938/consoleFull)
 for   PR 4144 at commit 
[`0a875ec`](https://github.com/apache/spark/commit/0a875ec4c4c272998fc0ebd6ddd782dc41e17970).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [WIP][SPARK-4586][MLLIB] Python API for ML pip...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/4151#issuecomment-70960645
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25935/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [WIP][SPARK-4586][MLLIB] Python API for ML pip...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4151#issuecomment-70960636
  
  [Test build #25935 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25935/consoleFull)
 for   PR 4151 at commit 
[`56de571`](https://github.com/apache/spark/commit/56de5717c258b90ea5b02a6e4263f365f39628f6).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `class PipelineStage(Params):`
  * `class Estimator(PipelineStage):`
  * `class Transformer(PipelineStage):`
  * `class Pipeline(Estimator):`
  * `class PipelineModel(Transformer):`
  * `class JavaWrapper(object):`
  * `class JavaEstimator(Estimator, JavaWrapper):`
  * `class JavaTransformer(Transformer, JavaWrapper):`
  * `class LogisticRegression(JavaEstimator, HasFeaturesCol, HasLabelCol, 
HasPredictionCol, HasMaxIter,`
  * `class LogisticRegressionModel(Transformer):`
  * `class Tokenizer(JavaTransformer, HasInputCol, HasOutputCol):`
  * `class HashingTF(JavaTransformer, HasInputCol, HasOutputCol):`
  * `class Param(object):`
  * `class Params(Identifiable):`
  * `return """class Has%s(Params):`
  * `class HasMaxIter(Params):`
  * `class HasRegParam(Params):`
  * `class HasFeaturesCol(Params):`
  * `class HasLabelCol(Params):`
  * `class HasPredictionCol(Params):`
  * `class HasInputCol(Params):`
  * `class HasOutputCol(Params):`
  * `class Identifiable(object):`



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5347][CORE] Change FileSplit to InputSp...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4150#issuecomment-70960330
  
  [Test build #25937 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25937/consoleFull)
 for   PR 4150 at commit 
[`9e04a54`](https://github.com/apache/spark/commit/9e04a547115bbcb4c19b55b451ca3afe09955e9f).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `  protected case class Keyword(str: String) `
  * `class SqlLexical extends StdLexical `



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5347][CORE] Change FileSplit to InputSp...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/4150#issuecomment-70960337
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25937/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4879] [WIP] Use driver to coordinate Ha...

[mccheah]

Github user mccheah commented on the pull request:

https://github.com/apache/spark/pull/4066#issuecomment-70959680
  
The linked pull request takes your ideas, makes them compatible with 
master, and adds unit tests. Feel free to take a look.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4879] Use the Spark driver to authorize...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4155#issuecomment-70959593
  
  [Test build #25940 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25940/consoleFull)
 for   PR 4155 at commit 
[`6b543ba`](https://github.com/apache/spark/commit/6b543bae5d9a6ea27408a57663986d3947ea7ab1).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4879] Use the Spark driver to authorize...

[mccheah]

GitHub user mccheah opened a pull request:

https://github.com/apache/spark/pull/4155

[SPARK-4879] Use the Spark driver to authorize Hadoop commits.

This is a version of https://github.com/apache/spark/pull/4066/ which is up 
to date with master and has unit tests.

Previously, SparkHadoopWriter always committed its tasks without question. 
The problem is that when speculation is turned on, sometimes this can result in 
multiple tasks committing their output to the same
partition. Even though an HDFS-writing task may be re-launched due to 
speculation, the original task is not killed and may eventually commit as well.

This can cause strange race conditions where multiple tasks that commit 
interfere with each other, with the result being that some partition files are 
actually lost entirely. For more context on these kinds of scenarios, see the 
aforementioned JIRA ticket.

In Hadoop MapReduce jobs, the application master is a central coordinator 
that authorizes whether or not any given task can commit. Before a task commits 
its output, it queries the application master as to whether or not such a 
commit is safe, and the application master does bookkeeping as tasks are 
requesting commits. Duplicate tasks that would write to files that were already 
written to from other tasks are prohibited from committing.

This patch emulates that functionality - the crucial missing component was 
a central arbitrator, which is now a module called the OutputCommitCoordinator. 
The coordinator lives on the driver and the executors can obtain a reference to 
this actor and request its permission to commit. As tasks commit and are 
reported as completed successfully or unsuccessfully by the DAGScheduler, the 
commit coordinator is informed of the task completion events as well to update 
its internal state.

Future work includes more rigorous unit testing and extra optimizations 
should this patch cause a performance regression. It is unclear what the 
overall cost of communicating back to the driver on every hadoop-committing 
task will be.

cc @MingyuKim @JoshRosen  @ash211 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mccheah/spark commit-coordination

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/4155.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4155


commit 6e6f748561c5c3b8fa08796d39553b83216a2f29
Author: mcheah 
Date:   2015-01-21T18:24:41Z

[SPARK-4879] Use the Spark driver to authorize Hadoop commits.

Previously, SparkHadoopWriter always committed its tasks without
question. The problem is that when speculation is turned on, sometimes
this can result in multiple tasks committing their output to the same
partition. Even though an HDFS-writing task may be re-launched due to
speculation, the original task is not killed and may eventually commit
as well.

This can cause strange race conditions where multiple tasks that commit
interfere with each other, with the result being that some partition
files are actually lost entirely. For more context on these kinds of
scenarios, see the aforementioned JIRA ticket.

In Hadoop MapReduce jobs, the application master is a central
coordinator that authorizes whether or not any given task can commit.
Before a task commits its output, it queries the application master as
to whether or not such a commit is safe, and the application master does
bookeeping as tasks are requesting commits. Duplicate tasks that would
write to files that were already written to from other tasks are
prohibited from committing.

This patch emulates that functionality - the crucial missing component was
a central arbitrator, which is now a module called the 
OutputCommitCoordinator.
The coordinator lives on the driver and the executors can obtain a reference
to this actor and request its permission to commit. As tasks commit and are
reported as completed successfully or unsuccessfully by the DAGScheduler,
the commit coordinator is informed of the task completion events as well
to update its internal state.

commit bc80770fea44474aba5e1ece57968e7c8c311f85
Author: mcheah 
Date:   2015-01-22T01:31:39Z

Unit tests for OutputCommitCoordinator

commit c9decc68377db246e720afe802a91852b043e364
Author: mcheah 
Date:   2015-01-22T01:34:05Z

Scalastyle fixes

commit 6b543bae5d9a6ea27408a57663986d3947ea7ab1
Author: mcheah 
Date:   2015-01-22T01:52:14Z

Removing redundant accumulator in unit test




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the featur

[GitHub] spark pull request: [MLLIB] SPARK-5362 (4526, 2372) Gradient and O...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/4152#issuecomment-70959231
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25936/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [MLLIB] SPARK-5362 (4526, 2372) Gradient and O...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4152#issuecomment-70959227
  
  [Test build #25936 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25936/consoleFull)
 for   PR 4152 at commit 
[`43c6fec`](https://github.com/apache/spark/commit/43c6fec8caa72a9d853284736fb81efe66a820cd).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5297][Streaming][backport] Backport SPA...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4154#issuecomment-70958240
  
  [Test build #25939 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25939/consoleFull)
 for   PR 4154 at commit 
[`a91f295`](https://github.com/apache/spark/commit/a91f295abee7aaf758c6c9829ab62e36ca0b0022).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5297][Streaming][backport] Backport SPA...

[jerryshao]

GitHub user jerryshao opened a pull request:

https://github.com/apache/spark/pull/4154

[SPARK-5297][Streaming][backport] Backport SPARK-5297 to Spark 1.2



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jerryshao/apache-spark SPARK-5297-backport

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/4154.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4154


commit de0c2a91b66d1a83b92516d8375c979b7b886e15
Author: jerryshao 
Date:   2015-01-19T06:49:00Z

Fix Java fileInputStream type erasure problem

commit a91f295abee7aaf758c6c9829ab62e36ca0b0022
Author: jerryshao 
Date:   2015-01-19T07:12:35Z

Add Mima exclusion

Conflicts:
project/MimaExcludes.scala




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5202] [SQL] Add hql variable substituti...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/4003


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5202] [SQL] Add hql variable substituti...

[rxin]

Github user rxin commented on the pull request:

https://github.com/apache/spark/pull/4003#issuecomment-70956868
  
Thanks. Merging in master!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5355] make SparkConf thread-safe

[davies]

Github user davies commented on the pull request:

https://github.com/apache/spark/pull/4143#issuecomment-70956714
  
I'm not sure. @pwendell What's your prefer? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5317]Set BoostingStrategy.defaultParams...

[mengxr]

Github user mengxr commented on a diff in the pull request:

https://github.com/apache/spark/pull/4103#discussion_r23349888
  
--- Diff: 
mllib/src/main/scala/org/apache/spark/mllib/tree/configuration/BoostingStrategy.scala
 ---
@@ -70,20 +70,31 @@ object BoostingStrategy {
 
   /**
* Returns default configuration for the boosting algorithm
+   * @param algoStr Learning goal.  Supported:
--- End diff --

This is my bad. In Scala, the parameter name is part of the API. So let's 
not rename `algo` to `algoStr`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5317]Set BoostingStrategy.defaultParams...

[mengxr]

Github user mengxr commented on a diff in the pull request:

https://github.com/apache/spark/pull/4103#discussion_r23349889
  
--- Diff: 
mllib/src/main/scala/org/apache/spark/mllib/tree/configuration/BoostingStrategy.scala
 ---
@@ -70,20 +70,31 @@ object BoostingStrategy {
 
   /**
* Returns default configuration for the boosting algorithm
+   * @param algoStr Learning goal.  Supported:
+   * 
[[org.apache.spark.mllib.tree.configuration.Algo.Classification]],
--- End diff --

It is clearer if we say `"Classification" and "Regression"` instead of list 
classes for `algo: String`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5202] [SQL] Add hql variable substituti...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4003#issuecomment-70956300
  
  [Test build #25934 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25934/consoleFull)
 for   PR 4003 at commit 
[`bb41fd6`](https://github.com/apache/spark/commit/bb41fd688c60e352eda5a0d6d934ffbf7f42bcae).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5202] [SQL] Add hql variable substituti...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/4003#issuecomment-70956306
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25934/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-5357: Update commons-codec version to 1....

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/4153#issuecomment-70956162
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5355] make SparkConf thread-safe

[JoshRosen]

Github user JoshRosen commented on the pull request:

https://github.com/apache/spark/pull/4143#issuecomment-70956097
  
@davies Whoops, sorry for not coordinating on this.  Should we switch to 
ConcurrentMap?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4939] move to next locality when no pen...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/3779#issuecomment-70955957
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25931/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4939] move to next locality when no pen...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/3779#issuecomment-70955949
  
  [Test build #25931 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25931/consoleFull)
 for   PR 3779 at commit 
[`414e79e`](https://github.com/apache/spark/commit/414e79eafc7e3b49b63bd7c626459a748c5ced02).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5355] make SparkConf thread-safe

[davies]

Github user davies commented on the pull request:

https://github.com/apache/spark/pull/4143#issuecomment-70955650
  
Just come up a new version using j.u.c.ConcurrentMap: 
https://github.com/davies/spark/commit/ae4d305eb96f8d4f9d08b08ba2c7f590b196494d


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-5357: Update commons-codec version to 1....

[MattWhelan]

GitHub user MattWhelan opened a pull request:

https://github.com/apache/spark/pull/4153

SPARK-5357: Update commons-codec version to 1.10 (current)

Resolves https://issues.apache.org/jira/browse/SPARK-5357

In commons-codec 1.5, Base64 instances are not thread safe.  That was only 
true from 1.4-1.6.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/MattWhelan/spark depsUpdate

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/4153.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4153


commit b4a91f478d496b48c03d28bc44a9a848b5e93c85
Author: Matt Whelan 
Date:   2015-01-22T01:19:54Z

SPARK-5357: Update commons-codec version to 1.10 (current)




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5317]Set BoostingStrategy.defaultParams...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4103#issuecomment-70955511
  
  [Test build #25933 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25933/consoleFull)
 for   PR 4103 at commit 
[`3b72875`](https://github.com/apache/spark/commit/3b728750151f40d4f7ea8168b2888b1dec1d10b9).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5317]Set BoostingStrategy.defaultParams...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/4103#issuecomment-70955515
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25933/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5147][Streaming] Delete the received da...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4149#issuecomment-70954372
  
  [Test build #25932 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25932/consoleFull)
 for   PR 4149 at commit 
[`c4cf067`](https://github.com/apache/spark/commit/c4cf067b31fc43229bd1977a9273b8f34a8fcd49).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5147][Streaming] Delete the received da...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/4149#issuecomment-70954379
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25932/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org