[GitHub] spark pull request: [SPARK-13636][SQL] Directly consume UnsafeRow ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11484#issuecomment-191642597
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52377/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13636][SQL] Directly consume UnsafeRow ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11484#issuecomment-191642595
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13036][SPARK-13318][SPARK-13319] Add sa...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11203#issuecomment-191642847
  
**[Test build #52381 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52381/consoleFull)**
 for PR 11203 at commit 
[`ecd1df1`](https://github.com/apache/spark/commit/ecd1df10b68a78ba6da19a3111f3ea4cf09a16a2).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13036][SPARK-13318][SPARK-13319] Add sa...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11203#issuecomment-191643227
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52381/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12720] [SQL] SQL Generation Support for...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11283#issuecomment-191643198
  
**[Test build #52373 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52373/consoleFull)**
 for PR 11283 at commit 
[`749be1b`](https://github.com/apache/spark/commit/749be1b2de07603a7fc9961872954c24a584a1fb).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13036][SPARK-13318][SPARK-13319] Add sa...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11203#issuecomment-191643226
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13495][SQL] Add Null Filters in the que...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11372#issuecomment-191643447
  
**[Test build #52383 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52383/consoleFull)**
 for PR 11372 at commit 
[`2a469e8`](https://github.com/apache/spark/commit/2a469e84a618ffe69580eaab6ca77e2a3ab1b7f7).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12720] [SQL] SQL Generation Support for...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11283#issuecomment-191643566
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52373/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12720] [SQL] SQL Generation Support for...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11283#issuecomment-191643564
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13640][SQL] Synchronize ScalaReflection...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11487#issuecomment-191643706
  
**[Test build #52375 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52375/consoleFull)**
 for PR 11487 at commit 
[`389d644`](https://github.com/apache/spark/commit/389d644f776504ee335cd26c121dc84c1902cf32).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13640][SQL] Synchronize ScalaReflection...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11487#issuecomment-191643939
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13640][SQL] Synchronize ScalaReflection...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11487#issuecomment-191643944
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52375/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13466][SQL] Remove projects that become...

[davies]

Github user davies commented on the pull request:

https://github.com/apache/spark/pull/11341#issuecomment-191644061
  
LGTM, merging this into master, thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10562][SQL] support mixed case partitio...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/9226#discussion_r54846913
  
--- Diff: 
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala ---
@@ -218,25 +233,19 @@ private[hive] class HiveMetastoreCatalog(val client: 
ClientInterface, hive: Hive
   }
 }
 
-val metastorePartitionColumns = userSpecifiedSchema.map { schema =>
-  val fields = partitionColumns.map(col => schema(col))
-  fields.map { field =>
-HiveColumn(
-  name = field.name,
-  hiveType = HiveMetastoreTypes.toMetastoreType(field.dataType),
-  comment = "")
-  }.toSeq
-}.getOrElse {
-  if (partitionColumns.length > 0) {
-// The table does not have a specified schema, which means that 
the schema will be inferred
-// when we load the table. So, we are not expecting partition 
columns and we will discover
-// partitions when we load the table. However, if there are 
specified partition columns,
-// we simply ignore them and provide a warning message.
-logWarning(
-  s"The schema and partitions of table $tableIdent will be 
inferred when it is loaded. " +
-s"Specified partition columns 
(${partitionColumns.mkString(",")}) will be ignored.")
+if (userSpecifiedSchema.isDefined && partitionColumns.length > 0) {
+  tableProperties.put("spark.sql.sources.schema.numPartCols", 
partitionColumns.length.toString)
+  partitionColumns.zipWithIndex.foreach { case (partCol, index) =>
+tableProperties.put(s"spark.sql.sources.schema.partCol.$index", 
partCol)
   }
-  Seq.empty[HiveColumn]
+} else {
+  // The table does not have a specified schema, which means that the 
schema will be inferred
+  // when we load the table. So, we are not expecting partition 
columns and we will discover
+  // partitions when we load the table. However, if there are 
specified partition columns,
+  // we simply ignore them and provide a warning message.
+  logWarning(
+s"The schema and partitions of table $tableIdent will be inferred 
when it is loaded. " +
+  s"Specified partition columns 
(${partitionColumns.mkString(",")}) will be ignored.")
--- End diff --

line 293 "partitionColumns = Nil" is after an asset: 
`assert(partitionColumns.isEmpty)`.
We can not write out partitioned table in a hive compatible way, so I think 
this PR is not the root cause that break your code.
But it's surprising that the written table is partitioned at hive side in 
1.5.2, can you double check it?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13543][SQL] Support for specifying comp...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11464#issuecomment-191644225
  
**[Test build #52384 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52384/consoleFull)**
 for PR 11464 at commit 
[`d2ace37`](https://github.com/apache/spark/commit/d2ace37bea1cd8a4c8e1cfb4b1b7fb2ace67c005).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13543][SQL] Support for specifying comp...

[HyukjinKwon]

Github user HyukjinKwon commented on the pull request:

https://github.com/apache/spark/pull/11464#issuecomment-191644382
  
test this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13466][SQL] Remove projects that become...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/11341


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10562][SQL] support mixed case partitio...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/9226#discussion_r54847179
  
--- Diff: 
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala ---
@@ -218,25 +233,19 @@ private[hive] class HiveMetastoreCatalog(val client: 
ClientInterface, hive: Hive
   }
 }
 
-val metastorePartitionColumns = userSpecifiedSchema.map { schema =>
-  val fields = partitionColumns.map(col => schema(col))
-  fields.map { field =>
-HiveColumn(
-  name = field.name,
-  hiveType = HiveMetastoreTypes.toMetastoreType(field.dataType),
-  comment = "")
-  }.toSeq
-}.getOrElse {
-  if (partitionColumns.length > 0) {
-// The table does not have a specified schema, which means that 
the schema will be inferred
-// when we load the table. So, we are not expecting partition 
columns and we will discover
-// partitions when we load the table. However, if there are 
specified partition columns,
-// we simply ignore them and provide a warning message.
-logWarning(
-  s"The schema and partitions of table $tableIdent will be 
inferred when it is loaded. " +
-s"Specified partition columns 
(${partitionColumns.mkString(",")}) will be ignored.")
+if (userSpecifiedSchema.isDefined && partitionColumns.length > 0) {
+  tableProperties.put("spark.sql.sources.schema.numPartCols", 
partitionColumns.length.toString)
+  partitionColumns.zipWithIndex.foreach { case (partCol, index) =>
+tableProperties.put(s"spark.sql.sources.schema.partCol.$index", 
partCol)
   }
-  Seq.empty[HiveColumn]
+} else {
+  // The table does not have a specified schema, which means that the 
schema will be inferred
+  // when we load the table. So, we are not expecting partition 
columns and we will discover
+  // partitions when we load the table. However, if there are 
specified partition columns,
+  // we simply ignore them and provide a warning message.
+  logWarning(
+s"The schema and partitions of table $tableIdent will be inferred 
when it is loaded. " +
+  s"Specified partition columns 
(${partitionColumns.mkString(",")}) will be ignored.")
--- End diff --

The root cause is line 270. Even in non-hive compatible mode, we still keep 
partition info in hive metadata before. This PR changed it and store partition 
info as properties.

We can't write out partitioned table in a hive-compatible way, so I think 
it's a safe change, you can't query that table at hive side anyway.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13543][SQL] Support for specifying comp...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11464#issuecomment-191646312
  
**[Test build #52385 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52385/consoleFull)**
 for PR 11464 at commit 
[`e4d4cfb`](https://github.com/apache/spark/commit/e4d4cfbc5a65ad95c4eee31ea5a2f09de7e5934a).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13637][SQL] use more information to sim...

[cloud-fan]

Github user cloud-fan commented on the pull request:

https://github.com/apache/spark/pull/11485#issuecomment-191646672
  
retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13637][SQL] use more information to sim...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11485#issuecomment-191648049
  
**[Test build #52386 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52386/consoleFull)**
 for PR 11485 at commit 
[`9db97ec`](https://github.com/apache/spark/commit/9db97ec3f641cba9b069821b456ed8a0dfbd905c).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13543][SQL] Support for specifying comp...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11464#issuecomment-191648669
  
**[Test build #52378 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52378/consoleFull)**
 for PR 11464 at commit 
[`2817c9a`](https://github.com/apache/spark/commit/2817c9a774f54c5f79828137bdb5a95c391c90c3).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13543][SQL] Support for specifying comp...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11464#issuecomment-191648913
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13543][SQL] Support for specifying comp...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11464#issuecomment-191648914
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52378/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13640][SQL] Synchronize ScalaReflection...

[srowen]

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/11487#issuecomment-191649365
  
I assume it still works in 2.11, and is at worst redundant but doesn't hurt 
much. This is definitely the right lock? Maybe a comment pointing to why that 
is correct would help since it isn't obvious


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13244][SQL][WIP] Prototyping: migrates ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11443#issuecomment-191649872
  
**[Test build #52387 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52387/consoleFull)**
 for PR 11443 at commit 
[`69695fb`](https://github.com/apache/spark/commit/69695fbe7e90bd78b608e116c483a670715fe916).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13432][SQL] add the source file name an...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11301#issuecomment-191654553
  
**[Test build #52388 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52388/consoleFull)**
 for PR 11301 at commit 
[`c763975`](https://github.com/apache/spark/commit/c763975f7f7a16503610a85ac9f5eb542f76cacf).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13244][SQL][WIP] Prototyping: migrates ...

[liancheng]

Github user liancheng commented on the pull request:

https://github.com/apache/spark/pull/11443#issuecomment-191661797
  
@rxin We don't. As what have been done in this PR, we only replaces 
`DataFrame` with `Dataset` on Java side.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [HOT-FIX] Recover some deprecations for 2.10 c...

[srowen]

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/11488#issuecomment-191664081
  
OK, I'll merge when tests pass here, thank you.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12154] Upgrade to Jersey 2

[srowen]

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/11223#issuecomment-191669921
  
This needs a few things -- a rebase of course. 

Next I'm concerned with dependency harmonization. For example I see 
differing versions of Jackson dependencies being pulled in. I think I expect to 
see both 1.x and 2.x dependencies, but not different versions of 1.x or 2.x. 
Likewise with Jersey, it seems like a few 1.x dependencies are still coming in. 
Figuring this out is the painful part.

 We also need to update LICENSE files to reflect new dependencies. 

More tests can't hurt; I don't know how to test the security aspect. Tests 
do in general basically exercise the web UI already.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [MINOR] Fix typos in comments and testcase nam...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11481#issuecomment-191670172
  
**[Test build #52369 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52369/consoleFull)**
 for PR 11481 at commit 
[`dd43417`](https://github.com/apache/spark/commit/dd434174c1f17bc0f26e0b3acbe9e21120d05265).
 * This patch **fails from timeout after a configured wait of \`250m\`**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [MINOR] Fix typos in comments and testcase nam...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11481#issuecomment-191670305
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [MINOR] Fix typos in comments and testcase nam...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11481#issuecomment-191670308
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52369/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [MINOR] Fix typos in comments and testcase nam...

[srowen]

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/11481#issuecomment-191671137
  
Jenkins, test this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [MINOR] Fix typos in comments and testcase nam...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11481#issuecomment-191672931
  
**[Test build #52389 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52389/consoleFull)**
 for PR 11481 at commit 
[`dd43417`](https://github.com/apache/spark/commit/dd434174c1f17bc0f26e0b3acbe9e21120d05265).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-12925. Improve HiveInspectors.unwrap for...

[srowen]

Github user srowen commented on a diff in the pull request:

https://github.com/apache/spark/pull/11477#discussion_r54853932
  
--- Diff: 
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala ---
@@ -320,9 +320,11 @@ private[hive] trait HiveInspectors {
   case hvoi: HiveCharObjectInspector =>
 UTF8String.fromString(hvoi.getPrimitiveJavaObject(data).getValue)
   case x: StringObjectInspector if x.preferWritable() =>
-// Text is in UTF-8 already. No need to convert again via 
fromString
+// Text is in UTF-8 already. No need to convert again via 
fromString. Copy bytes
 val wObj = x.getPrimitiveWritableObject(data)
-UTF8String.fromBytes(wObj.getBytes, 0, wObj.getLength)
+val result = new Array[Byte](wObj.getLength())
--- End diff --

You can clone arrays with `.clone()`. Better yet, `Text` provides a 
`copyBytes()` method to return a copy of the bytes anyway.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13495][SQL] Add Null Filters in the que...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11372#issuecomment-191678584
  
**[Test build #52383 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52383/consoleFull)**
 for PR 11372 at commit 
[`2a469e8`](https://github.com/apache/spark/commit/2a469e84a618ffe69580eaab6ca77e2a3ab1b7f7).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13495][SQL] Add Null Filters in the que...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11372#issuecomment-191678874
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13495][SQL] Add Null Filters in the que...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11372#issuecomment-191678876
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52383/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13038][SPARK-13388][PySpark] Add load/s...

[yanboliang]

Github user yanboliang commented on the pull request:

https://github.com/apache/spark/pull/11266#issuecomment-191680086
  
@yinxusen I notice that your change made ```Pipeline``` and 
```PipelineModel``` as the wrapper of peer Java object. But I vote to keep the 
exist behavior, ```Pipeline``` still should be a pure Python object. 
We do not need to bother a peer Java object for ```Pipeline```. Because 
most stages(Estimator/Transformer) have already supported save/load, so 
implementing ```Pipeline``` persistence at Python side is more simple than your 
change.
May be we should here others' thoughts. cc @mengxr @jkbradley  


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13637][SQL] use more information to sim...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11485#issuecomment-191680443
  
**[Test build #52386 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52386/consoleFull)**
 for PR 11485 at commit 
[`9db97ec`](https://github.com/apache/spark/commit/9db97ec3f641cba9b069821b456ed8a0dfbd905c).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13637][SQL] use more information to sim...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11485#issuecomment-191680718
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52386/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13637][SQL] use more information to sim...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11485#issuecomment-191680716
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13465] Add a task failure listener to T...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11478#issuecomment-191681417
  
**[Test build #2605 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2605/consoleFull)**
 for PR 11478 at commit 
[`4c6e791`](https://github.com/apache/spark/commit/4c6e791371b29adf86fbfcad1b65fd32310aba2d).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13117] [Web UI] WebUI should use the lo...

[devaraj-kavali]

GitHub user devaraj-kavali opened a pull request:

https://github.com/apache/spark/pull/11490

[SPARK-13117] [Web UI] WebUI should use the local ip not 0.0.0.0

## What changes were proposed in this pull request?

In WebUI, now Jetty Server starts with SPARK_PUBLIC_DNS config value if it
is configured otherwise it starts with default value as '0.0.0.0'.

It is continuation as per the closed PR 
https://github.com/apache/spark/pull/11133 for the JIRA SPARK-13117 and 
discussion in SPARK-13117.

## How was this patch tested?

This has been verified using the command 'netstat -tnlp | grep ' to 
check on which IP/hostname is binding with the below steps. 

In the below results, mentioned PID in the command is the corresponding 
process id.

 Without the patch changes, 
Web UI(Jetty Server) is not taking the value configured for 
SPARK_PUBLIC_DNS and it is listening to all the interfaces. 
## Master
```
[devaraj@stobdtserver2 sbin]$ netstat -tnlp | grep 3930
tcp6   0  0 :::8080 :::*LISTEN  
3930/java
```


## Worker
```
[devaraj@stobdtserver2 sbin]$ netstat -tnlp | grep 4090
tcp6   0  0 :::8081 :::*LISTEN  
4090/java
```

## History Server Process, 
```
[devaraj@stobdtserver2 sbin]$ netstat -tnlp | grep 2471
tcp6   0  0 :::18080:::*LISTEN  
2471/java
```
## Driver
```
[devaraj@stobdtserver2 spark-master]$ netstat -tnlp | grep 6556
tcp6   0  0 :::4040 :::*LISTEN  
6556/java
```


 With the patch changes

# i. With SPARK_PUBLIC_DNS configured
If the SPARK_PUBLIC_DNS is configured then all the processes Web UI(Jetty 
Server) is getting bind to the configured value.
## Master
```
[devaraj@stobdtserver2 sbin]$ netstat -tnlp | grep 1561
tcp6   0  0 x.x.x.x:8080   :::*LISTEN  
1561/java
```
## Worker 
```
[devaraj@stobdtserver2 sbin]$ netstat -tnlp | grep 2229
tcp6   0  0 x.x.x.x:8081   :::*LISTEN  
2229/java
```
## History Server
```
[devaraj@stobdtserver2 sbin]$ netstat -tnlp | grep 3747
tcp6   0  0 x.x.x.x:18080  :::*LISTEN  
3747/java
```
## Driver
```
[devaraj@stobdtserver2 spark-master]$ netstat -tnlp | grep 6013
tcp6   0  0 x.x.x.x:4040   :::*LISTEN  
6013/java
```

# ii. Without SPARK_PUBLIC_DNS configured
If the SPARK_PUBLIC_DNS is not configured then all the processes Web 
UI(Jetty Server) will start with the '0.0.0.0' as default value.
## Master
```
[devaraj@stobdtserver2 sbin]$ netstat -tnlp | grep 4573
tcp6   0  0 :::8080 :::*LISTEN  
4573/java
```


## Worker
```
[devaraj@stobdtserver2 sbin]$ netstat -tnlp | grep 4703
tcp6   0  0 :::8081 :::*LISTEN  
4703/java
```

## History Server
```
[devaraj@stobdtserver2 sbin]$ netstat -tnlp | grep 4846
tcp6   0  0 :::18080:::*LISTEN  
4846/java
```

## Driver
```
[devaraj@stobdtserver2 sbin]$ netstat -tnlp | grep 5437
tcp6   0  0 :::4040 :::*LISTEN  
5437/java
```

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/devaraj-kavali/spark SPARK-13117-v1

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/11490.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #11490


commit 1d736ff7053b0df7b42b34ae738b7a2873e718a7
Author: Devaraj K 
Date:   2016-03-03T09:00:24Z

[SPARK-13117] [Web UI] WebUI should use the local ip not 0.0.0.0

In WebUI, now Jetty Server starts with SPARK_PUBLIC_DNS config value if it
is configured otherwise it starts with default value as '0.0.0.0'.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13117] [Web UI] WebUI should use the lo...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11490#issuecomment-191681962
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [HOT-FIX] Recover some deprecations for 2.10 c...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11488#issuecomment-191683625
  
**[Test build #2606 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2606/consoleFull)**
 for PR 11488 at commit 
[`dd66979`](https://github.com/apache/spark/commit/dd66979d7b3b84e4bc136350dd0c2d85d40f2e73).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13036][SPARK-13318][SPARK-13319] Add sa...

[yanboliang]

Github user yanboliang commented on the pull request:

https://github.com/apache/spark/pull/11203#issuecomment-191683786
  
LGTM for me, cc @mengxr 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [HOT-FIX] Recover some deprecations for 2.10 c...

[srowen]

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/11488#issuecomment-191685036
  
Merged to master


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13423] [WIP] [CORE] [SQL] [STREAMING] S...

[srowen]

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/11292#issuecomment-191686172
  
Merged to master


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [HOT-FIX] Recover some deprecations for 2.10 c...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/11488


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13423] [WIP] [CORE] [SQL] [STREAMING] S...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/11292


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13543][SQL] Support for specifying comp...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11464#issuecomment-191690696
  
**[Test build #52385 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52385/consoleFull)**
 for PR 11464 at commit 
[`e4d4cfb`](https://github.com/apache/spark/commit/e4d4cfbc5a65ad95c4eee31ea5a2f09de7e5934a).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13543][SQL] Support for specifying comp...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11464#issuecomment-191691005
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52385/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13543][SQL] Support for specifying comp...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11464#issuecomment-191691003
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13646] [MLlib] QuantileDiscretizer coun...

[eliasah]

GitHub user eliasah opened a pull request:

https://github.com/apache/spark/pull/11491

[SPARK-13646] [MLlib] QuantileDiscretizer counts dataset twice in get…

## What changes were proposed in this pull request?

It avoids counting the dataframe twice.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/eliasah/spark quantile-discretizer-patch

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/11491.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #11491


commit 0f319edae6a1d0272073b6eb60fa54d5bd765f78
Author: Abou Haydar Elias 
Date:   2016-03-03T10:05:01Z

[SPARK-13646] [MLlib] QuantileDiscretizer counts dataset twice in 
getSampledInput.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13646] [MLlib] QuantileDiscretizer coun...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11491#issuecomment-191691667
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13244][SQL][WIP] Prototyping: migrates ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11443#issuecomment-191691807
  
**[Test build #52390 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52390/consoleFull)**
 for PR 11443 at commit 
[`2bc521a`](https://github.com/apache/spark/commit/2bc521aa98e6c6ff828a5f75db47048ca6b6baef).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13543][SQL] Support for specifying comp...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11464#issuecomment-191691777
  
**[Test build #52384 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52384/consoleFull)**
 for PR 11464 at commit 
[`d2ace37`](https://github.com/apache/spark/commit/d2ace37bea1cd8a4c8e1cfb4b1b7fb2ace67c005).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13646] [MLlib] QuantileDiscretizer coun...

[srowen]

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/11491#issuecomment-191691673
  
Jenkins test this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13543][SQL] Support for specifying comp...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11464#issuecomment-191692104
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13543][SQL] Support for specifying comp...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11464#issuecomment-191692108
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52384/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13492][MESOS] Configurable Mesos framew...

[dragos]

Github user dragos commented on a diff in the pull request:

https://github.com/apache/spark/pull/11369#discussion_r54859540
  
--- Diff: 
core/src/main/scala/org/apache/spark/deploy/mesos/MesosClusterDispatcherArguments.scala
 ---
@@ -44,7 +44,7 @@ private[mesos] class 
MesosClusterDispatcherArguments(args: Array[String], conf:
   port = value
   parse(tail)
 
-case ("--webui-port" | "-p") :: IntParam(value) :: tail =>
--- End diff --

I agree that `-p` would always be matched by the pattern above, `"--port" | 
"-p"`. So removing it here wouldn't have any user-visible effects.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13647][SQL] also check if numeric value...

[cloud-fan]

GitHub user cloud-fan opened a pull request:

https://github.com/apache/spark/pull/11492

[SPARK-13647][SQL] also check if numeric value is within allowed range in 
_verify_type

## What changes were proposed in this pull request?

This PR makes the `_verify_type` in `types.py` more strict, also check if 
numeric value is within allowed range.

## How was this patch tested?

newly added doc test.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cloud-fan/spark py-verify

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/11492.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #11492


commit 23222b6429b1696b3d6690da0904affa27e0416c
Author: Wenchen Fan 
Date:   2016-03-03T09:55:54Z

also check if numeric value is within allowed range in _verify_type




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13492][MESOS] Configurable Mesos framew...

[dragos]

Github user dragos commented on the pull request:

https://github.com/apache/spark/pull/11369#issuecomment-191692543
  
LGTM.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13647][SQL] also check if numeric value...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11492#issuecomment-191692842
  
**[Test build #52391 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52391/consoleFull)**
 for PR 11492 at commit 
[`23222b6`](https://github.com/apache/spark/commit/23222b6429b1696b3d6690da0904affa27e0416c).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13646] [MLlib] QuantileDiscretizer coun...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11491#issuecomment-191692846
  
**[Test build #52392 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52392/consoleFull)**
 for PR 11491 at commit 
[`0f319ed`](https://github.com/apache/spark/commit/0f319edae6a1d0272073b6eb60fa54d5bd765f78).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13583][CORE][STREAMING] Remove unused i...

[srowen]

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/11438#issuecomment-191692936
  
Merged to master


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13646] [MLlib] QuantileDiscretizer coun...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11491#issuecomment-191693151
  
**[Test build #52392 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52392/consoleFull)**
 for PR 11491 at commit 
[`0f319ed`](https://github.com/apache/spark/commit/0f319edae6a1d0272073b6eb60fa54d5bd765f78).
 * This patch **fails Scala style tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13646] [MLlib] QuantileDiscretizer coun...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11491#issuecomment-191693158
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52392/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13646] [MLlib] QuantileDiscretizer coun...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11491#issuecomment-191693154
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13423] [WIP] [CORE] [SQL] [STREAMING] S...

[srowen]

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/11292#issuecomment-191694216
  
Nuts. I also just got bitten by 2.10 compatibility in two changes:

```
[error] 
/home/jenkins/workspace/spark-master-compile-maven-scala-2.10/core/src/main/scala/org/apache/spark/ui/jobs/JobsTab.scala:32:
 value contains is not a member of 
Option[org.apache.spark.scheduler.SchedulingMode.SchedulingMode]
[error] jobProgresslistener.schedulingMode.contains(SchedulingMode.FAIR)
```

`Option.contains` is 2.11+ only. But really this change should just be 
backed out. Hotfix coming.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13449] Naive Bayes wrapper in SparkR

[yanboliang]

Github user yanboliang commented on the pull request:

https://github.com/apache/spark/pull/11486#issuecomment-191694969
  
Labels of ML ```NaiveBayesModel``` are sorted(FYI #7284), so we do not need 
to store it as member variable. Then it can pass the binary compatibility check.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13432][SQL] add the source file name an...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11301#issuecomment-191696160
  
**[Test build #52388 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52388/consoleFull)**
 for PR 11301 at commit 
[`c763975`](https://github.com/apache/spark/commit/c763975f7f7a16503610a85ac9f5eb542f76cacf).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13432][SQL] add the source file name an...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11301#issuecomment-191696676
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13432][SQL] add the source file name an...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11301#issuecomment-191696679
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52388/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13646] [MLlib] QuantileDiscretizer coun...

[eliasah]

Github user eliasah commented on the pull request:

https://github.com/apache/spark/pull/11491#issuecomment-191696680
  
I'm not sure why the *scala style tests" fails. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13423] [HOTFIX] Static analysis fixes f...

[srowen]

GitHub user srowen opened a pull request:

https://github.com/apache/spark/pull/11493

[SPARK-13423] [HOTFIX] Static analysis fixes for 2.x / fixed for Scala 2.10

## What changes were proposed in this pull request?

Fixes compile problem due to inadvertent use of `Option.contains`, only in 
Scala 2.11. The change should have been to replace `Option.exists(_ == x)` with 
`== Some(x)`. Replacing exists with contains only makes sense for collections. 
Replacing use of `Option.exists` still makes sense though as it's misleading.

## How was this patch tested?

Jenkins tests / compilation

(If this patch involves UI changes, please attach a screenshot; otherwise, 
remove this)


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/srowen/spark SPARK-13423.2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/11493.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #11493


commit 217dde3483f6f031c5a5a13fcb0809382f89c355
Author: Sean Owen 
Date:   2016-03-03T10:25:44Z

Fixes compile problem due to inadvertent use of Option.contains, only in 
Scala 2.11




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13640][SQL] Synchronize ScalaReflection...

[sarutak]

Github user sarutak commented on the pull request:

https://github.com/apache/spark/pull/11487#issuecomment-191696955
  
This PR can resolve the issue mentioned in 
[SPARK-10719](https://issues.apache.org/jira/browse/SPARK-10719) right?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13646] [MLlib] QuantileDiscretizer coun...

[srowen]

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/11491#issuecomment-191697011
  
It's in the test output -- stray space before the comma :)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13640][SQL] Synchronize ScalaReflection...

[sarutak]

Github user sarutak commented on the pull request:

https://github.com/apache/spark/pull/11487#issuecomment-191697316
  
@zsxwing who reported SPARK-10719.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13423] [HOTFIX] Static analysis fixes f...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11493#issuecomment-191698101
  
**[Test build #52393 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52393/consoleFull)**
 for PR 11493 at commit 
[`217dde3`](https://github.com/apache/spark/commit/217dde3483f6f031c5a5a13fcb0809382f89c355).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13647][SQL] also check if numeric value...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11492#issuecomment-191698467
  
**[Test build #52391 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52391/consoleFull)**
 for PR 11492 at commit 
[`23222b6`](https://github.com/apache/spark/commit/23222b6429b1696b3d6690da0904affa27e0416c).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13647][SQL] also check if numeric value...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11492#issuecomment-191698743
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52391/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13244][SQL][WIP] Prototyping: migrates ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11443#issuecomment-191699392
  
**[Test build #52387 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52387/consoleFull)**
 for PR 11443 at commit 
[`69695fb`](https://github.com/apache/spark/commit/69695fbe7e90bd78b608e116c483a670715fe916).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13244][SQL][WIP] Prototyping: migrates ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11443#issuecomment-191699744
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52387/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13244][SQL][WIP] Prototyping: migrates ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11443#issuecomment-191699741
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13646] [MLlib] QuantileDiscretizer coun...

[eliasah]

Github user eliasah commented on the pull request:

https://github.com/apache/spark/pull/11491#issuecomment-191699932
  
Ok, sorry this is my first PR. Should I correct, push and pr again ?



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13646] [MLlib] QuantileDiscretizer coun...

[srowen]

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/11491#issuecomment-191703162
  
Yes just push a tiny commit to the same branch


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13646] [MLlib] QuantileDiscretizer coun...

[eliasah]

Github user eliasah commented on the pull request:

https://github.com/apache/spark/pull/11491#issuecomment-191703696
  
Done !


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13432][SQL] add the source file name an...

[kiszk]

Github user kiszk commented on the pull request:

https://github.com/apache/spark/pull/11301#issuecomment-191720188
  
Addressed all of CRs


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13647][SQL] also check if numeric value...

[cloud-fan]

Github user cloud-fan commented on the pull request:

https://github.com/apache/spark/pull/11492#issuecomment-191726473
  
cc @yhuai @davies 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13647][SQL] also check if numeric value...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/11492#discussion_r54870946
  
--- Diff: python/pyspark/sql/types.py ---
@@ -1137,7 +1148,28 @@ def _verify_type(obj, dataType):
 if type(obj) not in _acceptable_types[_type]:
 raise TypeError("%s can not accept object %r in type %s" % 
(dataType, obj, type(obj)))
 
-if isinstance(dataType, ArrayType):
+if isinstance(dataType, ByteType):
+if obj < -128 or obj > 127:
+raise ValueError("object of ByteType out of range, got: %s" % 
obj)
+
+elif isinstance(dataType, ShortType):
+if obj < -32768 or obj > 32767:
+raise ValueError("object of ShortType out of range, got: %s" % 
obj)
+
+elif isinstance(dataType, IntegerType):
+if obj < -2147483648 or obj > 2147483647:
+raise ValueError("object of IntegerType out of range, got: %s" 
% obj)
+
+elif isinstance(dataType, FloatType):
+from math import isinf
+from struct import pack, unpack
+
+if not isinf(obj):
+f = unpack("f", pack("f", obj))[0]
--- End diff --

Here I try to convert the `obj` to binary string as a `float32`, and then 
convert it back. If it becomes an `inf`, then it means the `obj` can not fit in 
`float32`.

Is there a better way to check if a python float can fit in `float32`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13637][SQL] use more information to sim...

[cloud-fan]

Github user cloud-fan commented on the pull request:

https://github.com/apache/spark/pull/11485#issuecomment-191726573
  
retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13637][SQL] use more information to sim...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11485#issuecomment-191726855
  
**[Test build #52394 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52394/consoleFull)**
 for PR 11485 at commit 
[`9db97ec`](https://github.com/apache/spark/commit/9db97ec3f641cba9b069821b456ed8a0dfbd905c).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13244][SQL][WIP] Prototyping: migrates ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11443#issuecomment-191729769
  
**[Test build #52390 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52390/consoleFull)**
 for PR 11443 at commit 
[`2bc521a`](https://github.com/apache/spark/commit/2bc521aa98e6c6ff828a5f75db47048ca6b6baef).
 * This patch **fails PySpark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13244][SQL][WIP] Prototyping: migrates ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11443#issuecomment-191729946
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52390/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13244][SQL][WIP] Prototyping: migrates ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11443#issuecomment-191729943
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13599] [BUILD] remove transitive groovy...

[steveloughran]

Github user steveloughran commented on the pull request:

https://github.com/apache/spark/pull/11449#issuecomment-191741549
  
The risk is deserialization; Groovy CVE-2015-3253  shows how groovy < 2.4.4 
makes it straightforward to use a class in Groovy to run arbitrary shell 
commands on the destination. This has been show on Java ObjectStream and 
XStream, so assume Kryo is vulnerable too.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13423] [HOTFIX] Static analysis fixes f...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11493#issuecomment-191743759
  
**[Test build #52393 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52393/consoleFull)**
 for PR 11493 at commit 
[`217dde3`](https://github.com/apache/spark/commit/217dde3483f6f031c5a5a13fcb0809382f89c355).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org