[GitHub] spark pull request #22461: [SPARK-25453][SQL][TEST] OracleIntegrationSuite I...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/22461 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22461: [SPARK-25453][SQL][TEST] OracleIntegrationSuite IllegalA...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/22461 Thanks! Merged to master/2.4 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22593: [Streaming][DOC] Fix typo & format in DataStreamW...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/22593#discussion_r221497310
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/streaming/DataStreamWriter.scala
---
@@ -46,14 +46,16 @@ final class DataStreamWriter[T] private[sql](ds:
Dataset[T]) {
/**
* Specifies how data of a streaming DataFrame/Dataset is written to a
streaming sink.
- * - `OutputMode.Append()`: only the new rows in the streaming
DataFrame/Dataset will be
- *written to the sink
- * - `OutputMode.Complete()`: all the rows in the streaming
DataFrame/Dataset will be written
- * to the sink every time these is some
updates
- * - `OutputMode.Update()`: only the rows that were updated in the
streaming DataFrame/Dataset
+ *
+ *`OutputMode.Append()`: only the new rows in the streaming
DataFrame/Dataset will be
+ *written to the sink.
--- End diff --
I would just format this similarly with
https://github.com/apache/spark/blob/e06da95cd9423f55cdb154a2778b0bddf7be984c/sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala#L338-L366
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22573: [SPARK-25558][SQL] Pushdown predicates for nested fields...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22573 That's great! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22582: [SPARK-25505][SQL][FOLLOWUP] Fix for attributes c...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/22582 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22573: [SPARK-25558][SQL] Pushdown predicates for nested fields...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/22573 Updating `Filter` APIs sounds reasonable to me. This should be part of our data source API v2. cc @cloud-fan @rxin @rdblue --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22524: [SPARK-25497][SQL] Limit operation within whole stage co...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22524 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22524: [SPARK-25497][SQL] Limit operation within whole stage co...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22524 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96807/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22524: [SPARK-25497][SQL] Limit operation within whole stage co...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22524 **[Test build #96807 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96807/testReport)** for PR 22524 at commit [`ed2c269`](https://github.com/apache/spark/commit/ed2c26928bb9f4ecf634245331a9be366a0642d5). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22597: [SPARK-25579][SQL] Use quoted attribute names if needed ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22597 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3600/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22597: [SPARK-25579][SQL] Use quoted attribute names if needed ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22597 **[Test build #96810 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96810/testReport)** for PR 22597 at commit [`f6c3dca`](https://github.com/apache/spark/commit/f6c3dca65b85888392f8299cc5fc20f698c6afc5). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22597: [SPARK-25579][SQL] Use quoted attribute names if needed ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22597 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22597: [SPARK-25579][SQL] Use quoted attribute names if ...
GitHub user dongjoon-hyun opened a pull request:
https://github.com/apache/spark/pull/22597
[SPARK-25579][SQL] Use quoted attribute names if needed in pushed ORC
predicates
## What changes were proposed in this pull request?
This PR aims to fix an ORC performance regression at Spark 2.4.0 RCs from
Spark 2.3.2. Currently, for column names with `.`, the pushed predicates are
ignored.
**Test Data**
```scala
scala> val df = spark.range(Int.MaxValue).sample(0.2).toDF("col.with.dot")
scala> df.write.mode("overwrite").orc("/tmp/orc")
```
**Spark 2.3.2**
```scala
scala> spark.sql("set spark.sql.orc.impl=native")
scala> spark.sql("set spark.sql.orc.filterPushdown=true")
scala> spark.time(spark.read.orc("/tmp/orc").where("`col.with.dot` <
10").show)
++
|col.with.dot|
++
| 1|
| 8|
++
Time taken: 1486 ms
scala> spark.time(spark.read.orc("/tmp/orc").where("`col.with.dot` <
10").show)
++
|col.with.dot|
++
| 1|
| 8|
++
Time taken: 163 ms
```
**Spark 2.4.0 RC2**
```scala
scala> spark.time(spark.read.orc("/tmp/orc").where("`col.with.dot` <
10").show)
++
|col.with.dot|
++
| 1|
| 8|
++
Time taken: 4087 ms
scala> spark.time(spark.read.orc("/tmp/orc").where("`col.with.dot` <
10").show)
++
|col.with.dot|
++
| 1|
| 8|
++
Time taken: 1998 ms
```
**This PR**
```scala
scala> spark.time(spark.read.orc("/tmp/orc").where("`col.with.dot` <
10").show)
++
|col.with.dot|
++
| 1|
| 8|
++
Time taken: 2477 ms
scala> spark.time(spark.read.orc("/tmp/orc").where("`col.with.dot` <
10").show)
++
|col.with.dot|
++
| 1|
| 8|
++
Time taken: 253 ms
```
## How was this patch tested?
Pass the Jenkins with the existing test and manually performance test.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dongjoon-hyun/spark SPARK-25579
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22597.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22597
commit f6c3dca65b85888392f8299cc5fc20f698c6afc5
Author: Dongjoon Hyun
Date: 2018-10-01T04:33:04Z
[SPARK-25579][SQL] Use quoted attribute names if needed in pushed ORC
predicates
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22596: Fix lint failure in 2.2
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22596 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22596: Fix lint failure in 2.2
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22596 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96808/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22596: Fix lint failure in 2.2
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22596 **[Test build #96808 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96808/testReport)** for PR 22596 at commit [`c4b6920`](https://github.com/apache/spark/commit/c4b6920ed24fd6398da27b978380427e3b0cb62a). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22527: [SPARK-17952][SQL] Nested Java beans support in createDa...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22527 **[Test build #96809 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96809/testReport)** for PR 22527 at commit [`d8083cf`](https://github.com/apache/spark/commit/d8083cf6f28c2de9c2580c042734117e3374f986). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22527: [SPARK-17952][SQL] Nested Java beans support in createDa...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/22527 Jenkins, ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22596: Fix lint failure in 2.2
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22596 Can you link the JIRA https://issues.apache.org/jira/browse/SPARK-25576 ? Please see https://spark.apache.org/contributing.html --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22596: Fix lint failure in 2.2
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22596 **[Test build #96808 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96808/testReport)** for PR 22596 at commit [`c4b6920`](https://github.com/apache/spark/commit/c4b6920ed24fd6398da27b978380427e3b0cb62a). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22596: Fix lint failure in 2.2
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22596 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22539: [SPARK-25517][SQL] Detect/Infer date type in CSV file
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22539 Looks https://github.com/apache/spark/pull/21363 getting inactive. Can you take this over instead? You can pick up the commits there and open another PR. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22524: [SPARK-25497][SQL] Limit operation within whole stage co...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22524 **[Test build #96807 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96807/testReport)** for PR 22524 at commit [`ed2c269`](https://github.com/apache/spark/commit/ed2c26928bb9f4ecf634245331a9be366a0642d5). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22524: [SPARK-25497][SQL] Limit operation within whole stage co...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22524 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3599/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22524: [SPARK-25497][SQL] Limit operation within whole stage co...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22524 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22573: [SPARK-25558][SQL] Pushdown predicates for nested fields...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22573 Can we update public `Filter` API in Spark 3.0.0? @cloud-fan and @gatorsmile . --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22573: [SPARK-25558][SQL] Pushdown predicates for nested...
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/22573#discussion_r221477177
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
---
@@ -437,53 +436,65 @@ object DataSourceStrategy {
* @return a `Some[Filter]` if the input [[Expression]] is convertible,
otherwise a `None`.
*/
protected[sql] def translateFilter(predicate: Expression):
Option[Filter] = {
+// Recursively try to find an attribute name from the top level that
can be pushed down.
+def attrName(e: Expression): Option[String] = e match {
+ // In Spark and many data sources such as parquet, dots are used as
a column path delimiter;
+ // thus, we don't translate such expressions.
+ case a: Attribute if !a.name.contains(".") =>
+Some(a.name)
--- End diff --
Apache Spark 2.4.0 RC2 has a regression on this case. So, for now, this PR
doesn't have regssion on `master` branch.
```scala
scala> spark.time(spark.read.orc("/tmp/orc").where("`col.with.dot` =
5").count)
Time taken: 2405 ms
```
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22573: [SPARK-25558][SQL] Pushdown predicates for nested...
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/22573#discussion_r221476951
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
---
@@ -437,53 +436,65 @@ object DataSourceStrategy {
* @return a `Some[Filter]` if the input [[Expression]] is convertible,
otherwise a `None`.
*/
protected[sql] def translateFilter(predicate: Expression):
Option[Filter] = {
+// Recursively try to find an attribute name from the top level that
can be pushed down.
+def attrName(e: Expression): Option[String] = e match {
+ // In Spark and many data sources such as parquet, dots are used as
a column path delimiter;
+ // thus, we don't translate such expressions.
+ case a: Attribute if !a.name.contains(".") =>
+Some(a.name)
--- End diff --
Yes, @dbtsai . This PR has a regression on ORC at least. The following is
ORC result in Spark 2.3.2 and it will slowdown at least 5 times like Parquet.
> I know ORC doesn't work for now. We can have another followup PR to
address this.
```scala
scala> val df = spark.range(Int.MaxValue).sample(0.2).toDF("col.with.dot")
scala> df.write.mode("overwrite").orc("/tmp/orc")
scala> df.write.mode("overwrite").parquet("/tmp/parquet")
scala> spark.sql("set spark.sql.orc.impl=native")
scala> spark.sql("set spark.sql.orc.filterPushdown=true")
scala> spark.time(spark.read.orc("/tmp/orc").where("`col.with.dot` =
5").count)
Time taken: 803 ms
scala> spark.time(spark.read.parquet("/tmp/parquet").where("`col.with.dot`
= 5").count)
Time taken: 5573 ms
scala> spark.version
res6: String = 2.3.2
```
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22138: [SPARK-25151][SS] Apply Apache Commons Pool to KafkaData...
Github user HeartSaVioR commented on the issue: https://github.com/apache/spark/pull/22138 Kindly ask for reviewing. Please never mind when you're busy with fixing bugs on Spark 2.4 RC. @gaborgsomogyi I guess I left two things for committer decision: 1. define soft boundary and log when pooled objects exceed the boundary 2. documentation. Do you have more to review? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22565: [SPARK-25543][K8S] Print debug message iff execId...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/22565 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22565: [SPARK-25543][K8S] Print debug message iff execIdsRemove...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22565 Merged to master/branch-2.4. Thank you, @ScrapCodes and @liyinan926 . --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19410: [SPARK-22184][CORE][GRAPHX] GraphX fails in case of insu...
Github user szhem commented on the issue: https://github.com/apache/spark/pull/19410 Hello @mallman, @sujithjay, @felixcheung, @jkbradley, @mengxr, it's already about a year passed since this pull request has been opened. I'm just wondering whether there is any chance to get any feedback for this PR (understanding that all of you have a little or probably no time having your own more important activities) and get it either rejected or merged? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19373: [SPARK-22150][CORE] PeriodicCheckpointer fails in case o...
Github user szhem commented on the issue: https://github.com/apache/spark/pull/19373 Hello @sujithjay, @felixcheung, @jkbradley, @mengxr, it's already more than a year passed since this pull request has been opened. I'm just wondering whether there is any chance for this PR to be reviewed (understanding that all of you have a little or probably no time having your own more important activities) by someone and either rejected or merged. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21221: [SPARK-23429][CORE] Add executor memory metrics to heart...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21221 @mccheah When you merged the code, could you also leave the comments about which branches you did the merge? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21221: [SPARK-23429][CORE] Add executor memory metrics to heart...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21221 For the other reviewers, this was merged to master (not 2.4) --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22596: Fix lint failure in 2.2
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22596 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22596: Fix lint failure in 2.2
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22596 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22596: Fix lint failure in 2.2
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22596 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22596: Fix lint failure in 2.2
GitHub user samdvr opened a pull request: https://github.com/apache/spark/pull/22596 Fix lint failure in 2.2 ## What changes were proposed in this pull request? Line length fixes and ## How was this patch tested? Manually verified, but will ensure jenkins lint passes before merging You can merge this pull request into a Git repository by running: $ git pull https://github.com/samdvr/spark SPARK-25576 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/22596.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #22596 commit c4b6920ed24fd6398da27b978380427e3b0cb62a Author: Sam Davarnia <> Date: 2018-09-30T14:36:33Z fix liniting errors --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22539: [SPARK-25517][SQL] Detect/Infer date type in CSV file
Github user softmanu commented on the issue: https://github.com/apache/spark/pull/22539 @dongjoon-hyun @HyukjinKwon Hi, i was not well whole last week, now I am back, so yes, thanks for reviewing and all the comments. whether my PR is a duplicate or not we can see out later, all I worry here is the fact that **it's not working as expected**, and the whole steps of execution I have explained/captured at granular level in a very well structured and detailed manner so that it could be easy to understand, under this JIRA SPARK-25517 And sure, I will add a test case, and work upon it. P.S. I've found other different issues in spark same around date/timestamp which is not working at all because the implementation itself is missing totally. On this I will get back later, first let me resolve this current issue. Thanks, Manoranjan : ) --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22592: [SPARK-25575][WEBUI][SQL] SQL tab in the spark UI suppor...
Github user shahidki31 commented on the issue: https://github.com/apache/spark/pull/22592 Thank you @srowen . --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22577: [CORE][MINOR] Fix obvious error and compiling for...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/22577 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21669: [SPARK-23257][K8S] Kerberos Support for Spark on K8S
Github user suryag10 commented on the issue: https://github.com/apache/spark/pull/21669 > like it, but we could also first support cluster mode and add client mode after. Thats the reason i said "Point to note":)- --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22577: [CORE][MINOR] Fix obvious error and compiling for Scala ...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22577 Merged to master/2.4/2.3 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22594: [MINOR][SQL] When batch reading, the number of bytes can...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22594 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22594: [MINOR][SQL] When batch reading, the number of bytes can...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22594 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96806/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22594: [MINOR][SQL] When batch reading, the number of bytes can...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22594 **[Test build #96806 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96806/testReport)** for PR 22594 at commit [`e589e1e`](https://github.com/apache/spark/commit/e589e1ef83418a485c9d55a72209c0c86cf7b044). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22595: [SPARK-25577][Web UI] Add an on-off switch to display th...
Github user LantaoJin commented on the issue: https://github.com/apache/spark/pull/22595 If this PR could be merged, #22578 could be added as an additional column as well. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22595: [SPARK-25577][Web UI] Add an on-off switch to display th...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22595 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22595: [SPARK-25577][Web UI] Add an on-off switch to display th...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22595 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22595: [SPARK-25577][Web UI] Add an on-off switch to display th...
Github user LantaoJin commented on the issue: https://github.com/apache/spark/pull/22595 Gently ping @jerryshao @cloud-fan . Do you have a chance to review? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22595: [SPARK-25577][Web UI] Add an on-off switch to display th...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22595 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22595: [SPARK-25577][Web UI] Add an on-off switch to dis...
GitHub user LantaoJin opened a pull request:
https://github.com/apache/spark/pull/22595
[SPARK-25577][Web UI] Add an on-off switch to display the executor
additional columns
## What changes were proposed in this pull request?
[SPARK-17019](https://issues.apache.org/jira/browse/SPARK-17019) Expose
off-heap memory usage in WebUI. But it makes this additional columns hidden by
default. If we want to see them, we need change the css code to rebuild a
spark-core.jar. It's very inconvenient.
```
.on_heap_memory, .off_heap_memory {
display: none;
}
```
So I add an on-off switch to show those additional columns. And in future,
we don't afraid to add more columns.
## How was this patch tested?
https://user-images.githubusercontent.com/1853780/46256353-3a042700-c4dc-11e8-8b0f-305281d1fd17.png;>
https://user-images.githubusercontent.com/1853780/46256356-3f617180-c4dc-11e8-9d24-679db52343fe.png;>
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/LantaoJin/spark SPARK-25577
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22595.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22595
commit c758db01da06d0bc5f4466bab10a330ef812bcc6
Author: LantaoJin
Date: 2018-09-30T10:08:32Z
[SPARK-25577][Web UI] Add an on-off switch to display the executor
additional columns
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22594: [MINOR][SQL] When batch reading, the number of bytes can...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22594 **[Test build #96806 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96806/testReport)** for PR 22594 at commit [`e589e1e`](https://github.com/apache/spark/commit/e589e1ef83418a485c9d55a72209c0c86cf7b044). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22594: [MINOR][SQL] When batch reading, the number of bytes can...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22594 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22594: [MINOR][SQL] When batch reading, the number of bytes can...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22594 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3598/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22594: [MINOR][SQL] When batch reading, the number of by...
GitHub user 10110346 opened a pull request: https://github.com/apache/spark/pull/22594 [MINOR][SQL] When batch reading, the number of bytes can not be updated as expected. ## What changes were proposed in this pull request? When batch reading, the number of bytes can not be updated as expected. ## How was this patch tested? N/A You can merge this pull request into a Git repository by running: $ git pull https://github.com/10110346/spark inputMetrics Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/22594.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #22594 commit e589e1ef83418a485c9d55a72209c0c86cf7b044 Author: liuxian Date: 2018-09-30T09:14:20Z fix --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22581: [SPARK-25565][BUILD] Add scalastyle rule to check...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/22581 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22379: [SPARK-25393][SQL] Adding new function from_csv()
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/22379#discussion_r221449056
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala
---
@@ -520,7 +520,10 @@ object FunctionRegistry {
castAlias("date", DateType),
castAlias("timestamp", TimestampType),
castAlias("binary", BinaryType),
-castAlias("string", StringType)
+castAlias("string", StringType),
+
+// csv
+expression[CsvToStructs]("from_csv")
--- End diff --
This sounds a reasonable change. cc @rxin
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22581: [SPARK-25565][BUILD] Add scalastyle rule to check add Lo...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22581 Thanks for reviewing this @srowen and @wangyum --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22581: [SPARK-25565][BUILD] Add scalastyle rule to check add Lo...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22581 Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22592: [SPARK-25575][WEBUI][SQL] SQL tab in the spark UI suppor...
Github user shahidki31 commented on the issue: https://github.com/apache/spark/pull/22592 Thank you for the review @ajbozarth . --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
