[GitHub] spark issue #15340: [SPARKR][DOC] minor formatting and output cleanup for R ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15340
  
**[Test build #66304 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66304/consoleFull)**
 for PR 15340 at commit 
[`b9f47dd`](https://github.com/apache/spark/commit/b9f47dda68f08bfb5c2e9249efb238ce128c8905).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15340: [SPARKR][DOC] minor formatting and output cleanup...

[felixcheung]

GitHub user felixcheung opened a pull request:

https://github.com/apache/spark/pull/15340

[SPARKR][DOC] minor formatting and output cleanup for R vignettes

## What changes were proposed in this pull request?

Clean up output, format table, truncate long example output, hide warnings

(new - Left; existing - Right)

![image](https://cloud.githubusercontent.com/assets/8969467/19064018/5dcde4d0-89bc-11e6-857b-052df3f52a4e.png)

![image](https://cloud.githubusercontent.com/assets/8969467/19064034/6db09956-89bc-11e6-8e43-232d5c3fe5e6.png)

![image](https://cloud.githubusercontent.com/assets/8969467/19064058/88f09590-89bc-11e6-9993-61639e29dfdd.png)

![image](https://cloud.githubusercontent.com/assets/8969467/19064066/95ccbf64-89bc-11e6-877f-45af03ddcadc.png)

![image](https://cloud.githubusercontent.com/assets/8969467/19064082/a8445404-89bc-11e6-8532-26d8bc9b206f.png)


## How was this patch tested?

Run create-doc.sh manually


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/felixcheung/spark vignettes

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/15340.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #15340


commit b9f47dda68f08bfb5c2e9249efb238ce128c8905
Author: Felix Cheung 
Date:   2016-10-04T05:51:33Z

formatting and output fixes for vignettes




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14897: [SPARK-17338][SQL] add global temp view

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14897
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66298/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14897: [SPARK-17338][SQL] add global temp view

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14897
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14087: [SPARK-16411][SQL][STREAMING] Add textFile to Structured...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14087
  
**[Test build #66303 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66303/consoleFull)**
 for PR 14087 at commit 
[`25dfd09`](https://github.com/apache/spark/commit/25dfd09e194734f5d257041296c29dd79de81d1c).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14897: [SPARK-17338][SQL] add global temp view

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14897
  
**[Test build #66298 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66298/consoleFull)**
 for PR 14897 at commit 
[`cbbe122`](https://github.com/apache/spark/commit/cbbe122299a690cba7aff6c1a320d366513d42c9).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15102
  
**[Test build #3294 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3294/consoleFull)**
 for PR 15102 at commit 
[`a6c4970`](https://github.com/apache/spark/commit/a6c4970ace1df46e2d65c2cc8a606f3736454d35).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14087: [SPARK-16411][SQL][STREAMING] Add textFile to Str...

[ScrapCodes]

Github user ScrapCodes commented on a diff in the pull request:

https://github.com/apache/spark/pull/14087#discussion_r81689922
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/streaming/DataStreamReader.scala 
---
@@ -311,6 +311,37 @@ final class DataStreamReader 
private[sql](sparkSession: SparkSession) extends Lo
   @Experimental
   def text(path: String): DataFrame = format("text").load(path)
 
+  /**
+   * Loads text file(s) and returns a [[Dataset]] of String. The 
underlying schema of the Dataset
--- End diff --

I would like to be corrected, as I just followed the convention over here. 
Since this class does not have any vararg method for other APIs, I was doubtful 
in adding one myself.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15124: [SPARK-17559][MLLIB]persist edges if their storage level...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15124
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15124: [SPARK-17559][MLLIB]persist edges if their storage level...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15124
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66301/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15124: [SPARK-17559][MLLIB]persist edges if their storage level...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15124
  
**[Test build #66301 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66301/consoleFull)**
 for PR 15124 at commit 
[`de1c3e3`](https://github.com/apache/spark/commit/de1c3e3bbeadac3e0dc33154f25e7ae9523d085e).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14087: [SPARK-16411][SQL][STREAMING] Add textFile to Str...

[ScrapCodes]

Github user ScrapCodes commented on a diff in the pull request:

https://github.com/apache/spark/pull/14087#discussion_r81689547
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/streaming/DataStreamReader.scala 
---
@@ -21,13 +21,13 @@ import scala.collection.JavaConverters._
 
 import org.apache.spark.annotation.Experimental
 import org.apache.spark.internal.Logging
-import org.apache.spark.sql.{DataFrame, Dataset, SparkSession}
+import org.apache.spark.sql.{AnalysisException, DataFrame, Dataset, 
SparkSession}
 import org.apache.spark.sql.execution.datasources.DataSource
 import org.apache.spark.sql.execution.streaming.StreamingRelation
 import org.apache.spark.sql.types.StructType
 
 /**
- * Interface used to load a streaming [[Dataset]] from external storage 
systems (e.g. file systems,
+ * Class used to load a streaming [[Dataset]] from external storage 
systems (e.g. file systems,
--- End diff --

Understood, thanks for correcting !


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15337: [SPARK-17773] [Input/Output] Add VoidObjectInspector

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15337
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66299/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15337: [SPARK-17773] [Input/Output] Add VoidObjectInspector

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15337
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15337: [SPARK-17773] [Input/Output] Add VoidObjectInspector

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15337
  
**[Test build #66299 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66299/consoleFull)**
 for PR 15337 at commit 
[`ce0174f`](https://github.com/apache/spark/commit/ce0174f31cc6ca081a1b924fd465f2f37aaf59a5).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15325: [SPARK-17112][SQL] "select null" via JDBC triggers Illeg...

[dongjoon-hyun]

Github user dongjoon-hyun commented on the issue:

https://github.com/apache/spark/pull/15325
  
Thank You so much for review and merging, @rxin .


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15275: [SPARK-17702][SQL] Code generation including too ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/15275


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15275: [SPARK-17702][SQL] Code generation including too many mu...

[rxin]

Github user rxin commented on the issue:

https://github.com/apache/spark/pull/15275
  
Merging in master. Thanks.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15311: [SPARK-17721][MLlib][backport] Fix for multiplyin...

[bwahlgreen]

Github user bwahlgreen closed the pull request at:

https://github.com/apache/spark/pull/15311


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14828: [SPARK-17258][SQL] Parse scientific decimal literals as ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14828
  
**[Test build #66302 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66302/consoleFull)**
 for PR 14828 at commit 
[`a79e92f`](https://github.com/apache/spark/commit/a79e92f4d8d1e545bfb605d8ff33fece6ce66c0d).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14828: [SPARK-17258][SQL] Parse scientific decimal literals as ...

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/14828
  
retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14828: [SPARK-17258][SQL] Parse scientific decimal literals as ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14828
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14828: [SPARK-17258][SQL] Parse scientific decimal literals as ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14828
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66297/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14828: [SPARK-17258][SQL] Parse scientific decimal literals as ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14828
  
**[Test build #66297 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66297/consoleFull)**
 for PR 14828 at commit 
[`a79e92f`](https://github.com/apache/spark/commit/a79e92f4d8d1e545bfb605d8ff33fece6ce66c0d).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15292: [SPARK-17719][SQL] Unify and tie up options in a ...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/15292#discussion_r81685853
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRDD.scala
 ---
@@ -46,17 +45,18 @@ object JDBCRDD extends Logging {
* Takes a (schema, table) specification and returns the table's Catalyst
* schema.
*
-   * @param url - The JDBC url to fetch information from.
-   * @param table - The table name of the desired table.  This may also be 
a
-   *   SQL query wrapped in parentheses.
+   * @param options - JDBC options that contains url, table and other 
information.
*
* @return A StructType giving the table's Catalyst schema.
* @throws SQLException if the table specification is garbage.
* @throws SQLException if the table contains an unsupported type.
*/
-  def resolveTable(url: String, table: String, properties: Properties): 
StructType = {
+  def resolveTable(options: JDBCOptions): StructType = {
+val url = options.url
+val table = options.table
+val properties = options.asProperties
--- End diff --

`url`/`dbtable` are our Spark reserved option keys. To keep the external 
behaviors consistent, we should not change them. 

In addition, we should not pass them to the underlying JDBC drivers. That 
means, they should be consumed only by Spark. However, if the underlying JDBC 
drivers have such property key, users are not allowed to set them.

Let me know if you have any concern about it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15124: [SPARK-17559][MLLIB]persist edges if their storage level...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15124
  
**[Test build #66301 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66301/consoleFull)**
 for PR 15124 at commit 
[`de1c3e3`](https://github.com/apache/spark/commit/de1c3e3bbeadac3e0dc33154f25e7ae9523d085e).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15325: [SPARK-17112][SQL] "select null" via JDBC trigger...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/15325


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15325: [SPARK-17112][SQL] "select null" via JDBC triggers Illeg...

[rxin]

Github user rxin commented on the issue:

https://github.com/apache/spark/pull/15325
  
Thanks - merging in master/2.0.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15307: [WIP][SPARK-17731][SQL][STREAMING] Metrics for st...

[frreiss]

Github user frreiss commented on a diff in the pull request:

https://github.com/apache/spark/pull/15307#discussion_r81684775
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala
 ---
@@ -136,16 +139,30 @@ class StreamExecution(
   /** Whether the query is currently active or not */
   override def isActive: Boolean = state == ACTIVE
 
+  override def queryStatus: StreamingQueryInfo = {
+this.toInfo
+  }
+
   /** Returns current status of all the sources. */
   override def sourceStatuses: Array[SourceStatus] = {
 val localAvailableOffsets = availableOffsets
 sources.map(s =>
-  new SourceStatus(s.toString, 
localAvailableOffsets.get(s).map(_.toString))).toArray
+  new SourceStatus(
--- End diff --

Actually, you can probably drop most of the synchronization if you keep two 
`StreamMetrics` objects and preallocate the slots for counters. At least the 
way things are now, each counter in `StreamMetrics` is written once per batch. 
If you tweak `sourceStatuses()` to return the metrics from the most recent 
completed batch (i.e. the `StreamMetrics` object that's not currently being 
written to), there should be no overlap between readers and writers. Eventually 
you'll want to have more than one `StreamMetrics` object anyway, since the 
scheduler will need to pipeline multiple batches to reach latencies below the 
50-100ms level.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15246: [MINOR][SQL] Use resource path for test_script.sh

[weiqingy]

Github user weiqingy commented on the issue:

https://github.com/apache/spark/pull/15246
  
I have searched `src/test/resource `in the code base to get test cases 
which hard code `src/test/resources`. Except for two `ignore` tests in 
`SQLQuerySuite`, for those can not pass in IDE, they are modified to use 
resource path instead.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15307: [WIP][SPARK-17731][SQL][STREAMING] Metrics for structure...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15307
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66296/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15307: [WIP][SPARK-17731][SQL][STREAMING] Metrics for structure...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15307
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15307: [WIP][SPARK-17731][SQL][STREAMING] Metrics for structure...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15307
  
**[Test build #66296 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66296/consoleFull)**
 for PR 15307 at commit 
[`43e1ab1`](https://github.com/apache/spark/commit/43e1ab1df1406bf3ed7d9084c13bbe392b06b3b4).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15246: [MINOR][SQL] Use resource path for test_script.sh

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15246
  
**[Test build #66300 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66300/consoleFull)**
 for PR 15246 at commit 
[`6129187`](https://github.com/apache/spark/commit/6129187f21771088510b6acf0217f8d66a316ea5).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15102
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66289/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15102
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15102
  
**[Test build #66289 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66289/consoleFull)**
 for PR 15102 at commit 
[`a6c4970`](https://github.com/apache/spark/commit/a6c4970ace1df46e2d65c2cc8a606f3736454d35).
 * This patch **fails from timeout after a configured wait of \`250m\`**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15124: [SPARK-17559][MLLIB]persist edges if their storage level...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15124
  
**[Test build #3293 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3293/consoleFull)**
 for PR 15124 at commit 
[`bf94b4d`](https://github.com/apache/spark/commit/bf94b4dbcc4e8e0602715dce92f5053608674b43).
 * This patch passes all tests.
 * This patch **does not merge cleanly**.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15337: [SPARK-17773] [Input/Output] Add VoidObjectInspector

[seyfe]

Github user seyfe commented on the issue:

https://github.com/apache/spark/pull/15337
  
@hvanhovell. Thanks for the suggestion. 

Updated the HiveInspectorSuite, so 3 tests failed with below error:

`org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaVoidObjectInspector@7f485fda
 (of class 
org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaVoidObjectInspector)
scala.MatchError: 
org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaVoidObjectInspector@7f485fda
 (of class 
org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaVoidObjectInspector)
`

After applying the fix, all HiveInspectorSuite tests passed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15337: [SPARK-17773] [Input/Output] Add VoidObjectInspector

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15337
  
**[Test build #66299 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66299/consoleFull)**
 for PR 15337 at commit 
[`ce0174f`](https://github.com/apache/spark/commit/ce0174f31cc6ca081a1b924fd465f2f37aaf59a5).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14897: [SPARK-17338][SQL] add global temp view

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14897
  
**[Test build #66298 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66298/consoleFull)**
 for PR 14897 at commit 
[`cbbe122`](https://github.com/apache/spark/commit/cbbe122299a690cba7aff6c1a320d366513d42c9).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14897: [SPARK-17338][SQL] add global temp view

[cloud-fan]

Github user cloud-fan commented on the issue:

https://github.com/apache/spark/pull/14897
  
retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14828: [SPARK-17258][SQL] Parse scientific decimal literals as ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14828
  
**[Test build #66297 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66297/consoleFull)**
 for PR 14828 at commit 
[`a79e92f`](https://github.com/apache/spark/commit/a79e92f4d8d1e545bfb605d8ff33fece6ce66c0d).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15324: [SPARK-16872][ML] Gaussian Naive Bayes Classifier

[sethah]

Github user sethah commented on the issue:

https://github.com/apache/spark/pull/15324
  
Was there some discussion as to whether GaussianNB should be part of the 
NaiveBayes estimator or its own estimator? It seems the semantics are different 
enough between multinomial NB and Gaussian NB to at least warrant discussion. 
The meaning `theta` matrix in Gaussian NB vs multinomial is very different in 
this patch - one is matrix of Gaussian distribution paramters, and the other is 
a matrix of class conditional probabilities. Also, some params only apply to 
one and not the other. My apologies if I have missed this conversation 
somewhere. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15307: [WIP][SPARK-17731][SQL][STREAMING] Metrics for st...

[tdas]

Github user tdas commented on a diff in the pull request:

https://github.com/apache/spark/pull/15307#discussion_r81678871
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala
 ---
@@ -136,16 +139,30 @@ class StreamExecution(
   /** Whether the query is currently active or not */
   override def isActive: Boolean = state == ACTIVE
 
+  override def queryStatus: StreamingQueryInfo = {
+this.toInfo
+  }
+
   /** Returns current status of all the sources. */
   override def sourceStatuses: Array[SourceStatus] = {
 val localAvailableOffsets = availableOffsets
 sources.map(s =>
-  new SourceStatus(s.toString, 
localAvailableOffsets.get(s).map(_.toString))).toArray
+  new SourceStatus(
--- End diff --

yeah. you are probably right. Probably have to add synchronized to a lot of 
methods. :(


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15307: [WIP][SPARK-17731][SQL][STREAMING] Metrics for st...

[tdas]

Github user tdas commented on a diff in the pull request:

https://github.com/apache/spark/pull/15307#discussion_r81678796
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala
 ---
@@ -317,15 +358,18 @@ class StreamExecution(
 // TODO: Move this to IncrementalExecution.
 
 // Request unprocessed data from all sources.
-val newData = availableOffsets.flatMap {
-  case (source, available)
+val newData = timeIt(GET_BATCH_LATENCY) {
--- End diff --

Yeah. The intention in GET_BATCH_LATENCY is to measure the time taken in 
the non-lazy part.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15333: [SPARK-17761][SQL] Remove MutableRow

[liancheng]

Github user liancheng commented on the issue:

https://github.com/apache/spark/pull/15333
  
Would be nice to add a simple example to illustrate why we can't ensure 
that a `GenericInternalRow` is immutable. For example, for a 
`GenericInternalRow` with a `StructType` field, it's legal to put a 
`MutableRow` into the cell. This essentially makes the outer 
`GenericInternalRow` mutable. (In fact, we are already doing this in Spark, 
either intentionally or unintentionally.)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15322: [SPARK-17753][SQL] Allow a complex expression as ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/15322


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15322: [SPARK-17753][SQL] Allow a complex expression as the inp...

[hvanhovell]

Github user hvanhovell commented on the issue:

https://github.com/apache/spark/pull/15322
  
Merging to master/2.0. Thanks for the reviews!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15307: [WIP][SPARK-17731][SQL][STREAMING] Metrics for structure...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15307
  
**[Test build #66296 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66296/consoleFull)**
 for PR 15307 at commit 
[`43e1ab1`](https://github.com/apache/spark/commit/43e1ab1df1406bf3ed7d9084c13bbe392b06b3b4).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15304: Revert "[SPARK-17549][SQL] Revert Only collect table siz...

[yhuai]

Github user yhuai commented on the issue:

https://github.com/apache/spark/pull/15304
  
Changes look good. How about we change the title back to `[SPARK-17549] 
[SQL] Only collect table size stat in driver for cached relation`? Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14233: [SPARK-16490] [Examples] added a python example for chis...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14233
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66295/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14233: [SPARK-16490] [Examples] added a python example for chis...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14233
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14233: [SPARK-16490] [Examples] added a python example for chis...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14233
  
**[Test build #66295 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66295/consoleFull)**
 for PR 14233 at commit 
[`ca7cd78`](https://github.com/apache/spark/commit/ca7cd787e174e04fbe0fcdcff26c8169450abc7b).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15337: [SPARK-17773] [Input/Output] Add VoidObjectInspector

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15337
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66290/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15337: [SPARK-17773] [Input/Output] Add VoidObjectInspector

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15337
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15337: [SPARK-17773] [Input/Output] Add VoidObjectInspector

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15337
  
**[Test build #66290 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66290/consoleFull)**
 for PR 15337 at commit 
[`2c18d75`](https://github.com/apache/spark/commit/2c18d7553816517b0cb6df47023e622cf47e4766).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15314: [SPARK-17747][ML] WeightCol support non-double datatypes

[sethah]

Github user sethah commented on the issue:

https://github.com/apache/spark/pull/15314
  
@zhengruifeng We already test the label col using `checkNumericTypes` in 
`MLTestingUtils`. Temporary tests are ok with me but they still need to test 
every single numeric type, in each test suite. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15239: [SPARK-17665][SPARKR] Support options/mode all for read/...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15239
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66294/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15239: [SPARK-17665][SPARKR] Support options/mode all for read/...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15239
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15239: [SPARK-17665][SPARKR] Support options/mode all for read/...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15239
  
**[Test build #66294 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66294/consoleFull)**
 for PR 15239 at commit 
[`4126d04`](https://github.com/apache/spark/commit/4126d04befefd0cdf61deb608c01ada9248a8327).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15314: [SPARK-17747][ML] WeightCol support non-double datetypes

[zhengruifeng]

Github user zhengruifeng commented on the issue:

https://github.com/apache/spark/pull/15314
  
@jkbradley OK


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15314: [SPARK-17747][ML] WeightCol support non-double datetypes

[zhengruifeng]

Github user zhengruifeng commented on the issue:

https://github.com/apache/spark/pull/15314
  
@sethah I agreed that there should be an exhaustive helper. For now, I 
think some temporary tests may be enough.
By the way, we may also need to test all acceptable numerical datatypes for 
`LabelCol`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14233: [SPARK-16490] [Examples] added a python example for chis...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14233
  
**[Test build #66295 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66295/consoleFull)**
 for PR 14233 at commit 
[`ca7cd78`](https://github.com/apache/spark/commit/ca7cd787e174e04fbe0fcdcff26c8169450abc7b).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15314: [SPARK-17747][ML] WeightCol support non-double datetypes

[jkbradley]

Github user jkbradley commented on the issue:

https://github.com/apache/spark/pull/15314
  
@zhengruifeng Can you please update the PR title?  It says "datetypes" 
instead of "datatypes"  : )



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14233: [SPARK-16490] [Examples] added a python example for chis...

[jkbradley]

Github user jkbradley commented on the issue:

https://github.com/apache/spark/pull/14233
  
ok to test


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14653: [SPARK-10931][PYSPARK][ML] PySpark ML Models should cont...

[jkbradley]

Github user jkbradley commented on the issue:

https://github.com/apache/spark/pull/14653
  
ok to test

Sorry for the delay on this, but it'd be great to fix now!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15307: [WIP][SPARK-17731][SQL][STREAMING] Metrics for st...

[frreiss]

Github user frreiss commented on a diff in the pull request:

https://github.com/apache/spark/pull/15307#discussion_r81672432
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala
 ---
@@ -317,15 +358,18 @@ class StreamExecution(
 // TODO: Move this to IncrementalExecution.
 
 // Request unprocessed data from all sources.
-val newData = availableOffsets.flatMap {
-  case (source, available)
+val newData = timeIt(GET_BATCH_LATENCY) {
--- End diff --

Note that the time interval being measured here will have different 
semantics for different sources, depending on how much computation occurs 
inside the source's `getBatch` method vs. lazily when the data is read from the 
resulting Dataframe.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15307: [WIP][SPARK-17731][SQL][STREAMING] Metrics for st...

[frreiss]

Github user frreiss commented on a diff in the pull request:

https://github.com/apache/spark/pull/15307#discussion_r81672040
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala
 ---
@@ -136,16 +139,30 @@ class StreamExecution(
   /** Whether the query is currently active or not */
   override def isActive: Boolean = state == ACTIVE
 
+  override def queryStatus: StreamingQueryInfo = {
+this.toInfo
+  }
+
   /** Returns current status of all the sources. */
   override def sourceStatuses: Array[SourceStatus] = {
 val localAvailableOffsets = availableOffsets
 sources.map(s =>
-  new SourceStatus(s.toString, 
localAvailableOffsets.get(s).map(_.toString))).toArray
+  new SourceStatus(
--- End diff --

If this method is intended to be called from threads other than the 
scheduler thread, then the entire map really ought to be synchronized on 
`streamMetrics`'s lock. Otherwise this method could return a mixture of 
statistics from different points of time, even within a single source.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15333: [SPARK-17761][SQL] Remove MutableRow

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15333
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66286/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15333: [SPARK-17761][SQL] Remove MutableRow

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15333
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15333: [SPARK-17761][SQL] Remove MutableRow

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15333
  
**[Test build #66286 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66286/consoleFull)**
 for PR 15333 at commit 
[`09d533a`](https://github.com/apache/spark/commit/09d533adb3cd65de5017d3805ab92a92bc5a408f).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15239: [SPARK-17665][SPARKR] Support options/mode all for read/...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15239
  
**[Test build #66294 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66294/consoleFull)**
 for PR 15239 at commit 
[`4126d04`](https://github.com/apache/spark/commit/4126d04befefd0cdf61deb608c01ada9248a8327).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15339: Branch 2.0

[jkbradley]

Github user jkbradley commented on the issue:

https://github.com/apache/spark/pull/15339
  
I assume this is a mistake?  Please close this issue or fix it.  Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15339: Branch 2.0

[dongjoon-hyun]

Github user dongjoon-hyun commented on the issue:

https://github.com/apache/spark/pull/15339
  
Hi, @yashbopardikar .
Could you close this PR? It seems wrong. :)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15239: [SPARK-17665][SPARKR] Support options/mode all for read/...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/15239
  
Thanks, I missed the comment. I just addressed them.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15239: [SPARK-17665][SPARKR] Support options/mode all fo...

[HyukjinKwon]

Github user HyukjinKwon commented on a diff in the pull request:

https://github.com/apache/spark/pull/15239#discussion_r81672198
  
--- Diff: R/pkg/R/generics.R ---
@@ -651,23 +651,25 @@ setGeneric("write.jdbc", function(x, url, tableName, 
mode = "error", ...) {
 
 #' @rdname write.json
 #' @export
-setGeneric("write.json", function(x, path) { standardGeneric("write.json") 
})
+setGeneric("write.json", function(x, path, mode = NULL, ...) { 
standardGeneric("write.json") })
--- End diff --

Oh, yes, sure.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15102
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15102
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66288/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15102
  
**[Test build #66288 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66288/consoleFull)**
 for PR 15102 at commit 
[`7ff1059`](https://github.com/apache/spark/commit/7ff10599fdadcbdd2515b3216d35307e906de184).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15124: [SPARK-17559][MLLIB]persist edges if their storage level...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15124
  
**[Test build #3293 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3293/consoleFull)**
 for PR 15124 at commit 
[`bf94b4d`](https://github.com/apache/spark/commit/bf94b4dbcc4e8e0602715dce92f5053608674b43).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15124: [SPARK-17559][MLLIB]persist edges if their storage level...

[jkbradley]

Github user jkbradley commented on the issue:

https://github.com/apache/spark/pull/15124
  
LGTM
Will merge after re-running tests



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15328: [SPARKR][SPARK-17762] invokeJava fails when serialized a...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15328
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15328: [SPARKR][SPARK-17762] invokeJava fails when serialized a...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15328
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66293/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15328: [SPARKR][SPARK-17762] invokeJava fails when serialized a...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15328
  
**[Test build #66293 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66293/consoleFull)**
 for PR 15328 at commit 
[`9e764a5`](https://github.com/apache/spark/commit/9e764a5ce74cca1e816dac6a0b88a753578410ab).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15328: [SPARKR][SPARK-17762] invokeJava fails when serialized a...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15328
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66291/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15328: [SPARKR][SPARK-17762] invokeJava fails when serialized a...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15328
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15144: [SPARK-17587][PYTHON][MLLIB] SparseVector __getit...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/15144


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15328: [SPARKR][SPARK-17762] invokeJava fails when serialized a...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15328
  
**[Test build #66291 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66291/consoleFull)**
 for PR 15328 at commit 
[`7adc9b6`](https://github.com/apache/spark/commit/7adc9b6e1e0a16b14f29a4154646b4677a45fc2e).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15144: [SPARK-17587][PYTHON][MLLIB] SparseVector __getitem__ sh...

[jkbradley]

Github user jkbradley commented on the issue:

https://github.com/apache/spark/pull/15144
  
LGTM, merging with master and branch-2.0
Thank you @zero323 for the PR and @BryanCutler for reviewing !


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15144: [SPARK-17587][PYTHON][MLLIB] SparseVector __getitem__ sh...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15144
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66292/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15144: [SPARK-17587][PYTHON][MLLIB] SparseVector __getitem__ sh...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15144
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15144: [SPARK-17587][PYTHON][MLLIB] SparseVector __getitem__ sh...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15144
  
**[Test build #66292 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66292/consoleFull)**
 for PR 15144 at commit 
[`4162b06`](https://github.com/apache/spark/commit/4162b06c6e9aed079f0af90c8ba218b3371238e7).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15328: [SPARKR][SPARK-17762] invokeJava fails when serialized a...

[shivaram]

Github user shivaram commented on the issue:

https://github.com/apache/spark/pull/15328
  
Hmm - wont each object need to be deserialized on the Java side ? Will that 
object deserialize succeed ? Or to put it another way, can we add an end-to-end 
test that will exercise this code path ? 

It'll need to use 2G of memory I guess, so we might not want to run Jenkins 
on it each time but if we can manually verify it, it would make me feel better 
about us not missing something here.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14087: [SPARK-16411][SQL][STREAMING] Add textFile to Str...

[jodersky]

Github user jodersky commented on a diff in the pull request:

https://github.com/apache/spark/pull/14087#discussion_r81669340
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/streaming/DataStreamReader.scala 
---
@@ -311,6 +311,37 @@ final class DataStreamReader 
private[sql](sparkSession: SparkSession) extends Lo
   @Experimental
   def text(path: String): DataFrame = format("text").load(path)
 
+  /**
+   * Loads text file(s) and returns a [[Dataset]] of String. The 
underlying schema of the Dataset
--- End diff --

Should text files be plural here? The api would be more intuitive by 
copying the non-streaming equivalent with a vararg-method for multiple 
parameters


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15307: [WIP][SPARK-17731][SQL][STREAMING] Metrics for st...

[frreiss]

Github user frreiss commented on a diff in the pull request:

https://github.com/apache/spark/pull/15307#discussion_r81668841
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StatefulAggregate.scala
 ---
@@ -56,7 +57,12 @@ case class StateStoreRestoreExec(
 child: SparkPlan)
   extends execution.UnaryExecNode with StatefulOperator {
 
+  override lazy val metrics = Map(
+"numOutputRows" -> SQLMetrics.createMetric(sparkContext, "number of 
output rows"))
--- End diff --

The metric names should probably be in a separate, centralized list of 
constants. Users will want a single place in the API docs to find a list of all 
available metrics, and the list is likely to change quite frequently as 
Structured Streaming evolves.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15328: [SPARKR][SPARK-17762] invokeJava fails when serialized a...

[falaki]

Github user falaki commented on the issue:

https://github.com/apache/spark/pull/15328
  
@shivaram added unit tests. 
On Java Array limitation, we deserialize all of the arguments as one 
Array[Object]. So if even one of the arguments is larger than `INT_MAX` we will 
fail on the R side. But with this patch we can still handle those. There is 
still a problem when number of arguments is larger than INT_MAX. Am I missing 
another case?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15328: [SPARKR][SPARK-17762] invokeJava fails when serialized a...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15328
  
**[Test build #66293 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66293/consoleFull)**
 for PR 15328 at commit 
[`9e764a5`](https://github.com/apache/spark/commit/9e764a5ce74cca1e816dac6a0b88a753578410ab).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15339: Branch 2.0

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15339
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15339: Branch 2.0

[yashbopardikar]

GitHub user yashbopardikar opened a pull request:

https://github.com/apache/spark/pull/15339

Branch 2.0

## What changes were proposed in this pull request?

(Please fill in changes proposed in this fix)


## How was this patch tested?

(Please explain how this patch was tested. E.g. unit tests, integration 
tests, manual tests)


(If this patch involves UI changes, please attach a screenshot; otherwise, 
remove this)



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/apache/spark branch-2.0

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/15339.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #15339


commit 0fb01496c09defa1436dbb7f5e1cbc5461617a31
Author: WangTaoTheTonic 
Date:   2016-08-11T22:09:23Z

[SPARK-17022][YARN] Handle potential deadlock in driver handling messages

## What changes were proposed in this pull request?

We directly send RequestExecutors to AM instead of transfer it to 
yarnShedulerBackend first, to avoid potential deadlock.

## How was this patch tested?

manual tests

Author: WangTaoTheTonic 

Closes #14605 from WangTaoTheTonic/lock.

(cherry picked from commit ea0bf91b4a2ca3ef472906e50e31fd6268b6f53e)
Signed-off-by: Marcelo Vanzin 

commit b4047fc21cefcf6a43c1ee88af330a042f02bebc
Author: Dongjoon Hyun 
Date:   2016-08-12T06:40:12Z

[SPARK-16975][SQL] Column-partition path starting '_' should be handled 
correctly

Currently, Spark ignores path names starting with underscore `_` and `.`. 
This causes read-failures for the column-partitioned file data sources whose 
partition column names starts from '_', e.g. `_col`.

**Before**
```scala
scala> spark.range(10).withColumn("_locality_code", 
$"id").write.partitionBy("_locality_code").save("/tmp/parquet")
scala> spark.read.parquet("/tmp/parquet")
org.apache.spark.sql.AnalysisException: Unable to infer schema for 
ParquetFormat at /tmp/parquet20. It must be specified manually;
```

**After**
```scala
scala> spark.range(10).withColumn("_locality_code", 
$"id").write.partitionBy("_locality_code").save("/tmp/parquet")
scala> spark.read.parquet("/tmp/parquet")
res2: org.apache.spark.sql.DataFrame = [id: bigint, _locality_code: int]
```

Pass the Jenkins with a new test case.

Author: Dongjoon Hyun 

Closes #14585 from dongjoon-hyun/SPARK-16975-PARQUET.

(cherry picked from commit abff92bfdc7d4c9d2308794f0350561fe0ceb4dd)
Signed-off-by: Cheng Lian 

commit bde94cd71086fd348f3ba96de628d6df3f87dba5
Author: petermaxlee 
Date:   2016-08-12T06:56:55Z

[SPARK-17013][SQL] Parse negative numeric literals

## What changes were proposed in this pull request?
This patch updates the SQL parser to parse negative numeric literals as 
numeric literals, instead of unary minus of positive literals.

This allows the parser to parse the minimal value for each data type, e.g. 
"-32768S".

## How was this patch tested?
Updated test cases.

Author: petermaxlee 

Closes #14608 from petermaxlee/SPARK-17013.

(cherry picked from commit 00e103a6edd1a1f001a94d41dd1f7acc40a1e30f)
Signed-off-by: Reynold Xin 

commit 38378f59f2c91a6f07366aa2013522c334066c69
Author: Jagadeesan 
Date:   2016-08-13T10:25:03Z

[SPARK-12370][DOCUMENTATION] Documentation should link to examples …

## What changes were proposed in this pull request?

When documentation is built is should reference examples from the same 
build. There are times when the docs have links that point to files in the 
GitHub head which may not be valid on the current release. Changed that in URLs 
to make them point to the right tag in git using ```SPARK_VERSION_SHORT```

…from its own release version] [Streaming programming guide]

Author: Jagadeesan 

Closes #14596 from jagadeesanas2/SPARK-12370.

(cherry picked from commit e46cb78b3b9fd04a50b5ae50f360db612d656a48)
Signed-off-by: Sean Owen 

commit a21ecc9964bbd6e41a5464dcc85db1529de14d67
Author: Luciano Resende 
Date:   2016-08-13T10:42:38Z

[SPARK-17023][BUILD] Upgrade to Kafka 0.10.0.1 release

## What changes were proposed in this pull request?
Update Kafka streaming connector to use Kafka 0.10.0.1 release

## How was this patch tested?
Tested via Spark unit and integration tests

Author: